Medical scan diagnosing system

ABSTRACT

A medical scan diagnosing system is operable to receive a medical scan. Diagnosis data of the medical scan is generated by performing a medical scan inference function on the medical scan. The first medical scan is transmitted to a first client device associated with a user of the medical scan diagnosing system in response to the diagnosis data indicating that the medical scan corresponds to a non-normal diagnosis. The medical scan is displayed to the user via an interactive interface displayed by a display device corresponding to the first client device. Review data is received from the first client device, where the review data is generated by the first client device in response to a prompt via the interactive interface. Updated diagnosis data is generated based on the review data. The updated diagnosis data is transmitted to a second client device associated with a requesting entity.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility Patent Applications claims priority pursuant to35 U.S.C. § 119(e) to U.S. Provisional Application No. 62/511,150,entitled “MEDICAL SCAN ASSISTED REVIEW SYSTEM AND METHODS”, filed May25, 2017, which is hereby incorporated herein by reference in itsentirety and made part of the present U.S. Utility Patent Applicationfor all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND Technical Field

This invention relates generally to medical imaging devices andknowledge-based systems used in conjunction with client/server networkarchitectures.

Description of Related Art

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment;

FIG. 2A is a schematic block diagram of a client device in accordancewith various embodiments;

FIG. 2B is a schematic block diagram of one or more subsystems inaccordance with various embodiments;

FIG. 3 is a schematic block diagram of a database storage system inaccordance with various embodiments;

FIG. 4A is schematic block diagram of a medical scan entry in accordancewith various embodiments;

FIG. 4B is a schematic block diagram of abnormality data in accordancewith various embodiments;

FIG. 5 is a schematic block diagram of a user profile entry inaccordance with various embodiments;

FIG. 6 is a schematic block diagram of a medical scan analysis functionentry in accordance with various embodiments;

FIG. 7 is a schematic block diagram of an interface feature entry inaccordance with various embodiments;

FIG. 8A is a schematic block diagram of a medical scan assisted reviewsystem in accordance with various embodiments;

FIGS. 8B-8S are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIGS. 8T-8Y are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIGS. 9A-9B are schematic block diagrams of a medical scan reportlabeling system in accordance with various embodiments;

FIGS. 10A-10B are schematic block diagrams of a medical scan annotatingsystem in accordance with various embodiments;

FIGS. 10C-10V are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIGS. 11A-11C are schematic block diagram of a medical scan diagnosingsystem in accordance with various embodiments;

FIG. 12A is a schematic block diagram of a medical scan interfacefeature evaluator system in accordance with various embodiments;

FIGS. 12B-12C are graphical illustrations of an example interactiveinterface displayed on a client device in conjunction with variousembodiments;

FIG. 13A is a graphical illustration of a three-dimensional subregion inaccordance with various embodiments;

FIG. 13B is a graphical illustration of probability density functions inaccordance with various embodiments;

FIG. 13C is a graphical illustration a probability density function inaccordance with various embodiments;

FIG. 13D is a flowchart representation of an inference step inaccordance with various embodiments;

FIG. 13E is a flowchart representation of a detection step in accordancewith various embodiments.

FIG. 14A-14B are schematic block diagrams of a medical scan naturallanguage analysis system in accordance with various embodiments.

FIGS. 14C-14D are example input and output of a medical scan naturallanguage analysis system in accordance with various embodiments.

FIG. 15 is a schematic block diagram of a medical scan comparison systemin accordance with various embodiments;

FIG. 16 is a flowchart representation of a method in accordance with anembodiment.

FIG. 17 is a flowchart representation of a method in accordance with anembodiment.

FIG. 18 is a flowchart representation of a method in accordance with anembodiment.

FIG. 19 is a flowchart representation of a method in accordance with anembodiment.

FIG. 20 is a flowchart representation of a method in accordance with anembodiment.

FIG. 21 is a flowchart representation of a method in accordance with anembodiment.

FIG. 22 is a flowchart representation of a method in accordance with anembodiment.

FIG. 23 is a flowchart representation of a method in accordance with anembodiment.

FIG. 24 is a flowchart representation of a method in accordance with anembodiment.

FIG. 25 is a flowchart representation of a method in accordance with anembodiment.

DETAILED DESCRIPTION

FIG. 1 presents a medical scan processing system 100, which can includeone or more medical scan subsystems 101 that communicate bidirectionallywith one or more client devices 120 via a wired and/or wireless network150. The medical scan subsystems 101 can include a medical scan assistedreview system 102, medical scan report labeling system 104, a medicalscan annotator system 106, a medical scan diagnosing system 108, amedical scan interface feature evaluator system 110, a medical scanimage analysis system 112, a medical scan natural language analysissystem 114, and/or a medical scan comparison system 116. Some or all ofthe subsystems 101 can utilize the same processing devices, memorydevices, and/or network interfaces, for example, running on a same setof shared servers connected to network 150. Alternatively or inaddition, some or all of the subsystems 101 be assigned their ownprocessing devices, memory devices, and/or network interfaces, forexample, running separately on different sets of servers connected tonetwork 150. Some or all of the subsystems 101 can interact directlywith each other, for example, where one subsystem's output istransmitted directly as input to another subsystem via network 150.Network 150 can include one or more wireless and/or wired communicationsystems; one or more non-public intranet systems and/or public internetsystems; and/or one or more local area networks (LAN) and/or wide areanetworks (WAN).

The medical scan processing system 100 can further include a databasestorage system 140, which can include one or more servers, one or morememory devices of one or more subsystems 101, and/or one or more othermemory devices connected to network 150. The database storage system 140can store one or more shared databases and/or one or more files storedon one or more memory devices that include database entries as describedherein. The shared databases and/or files can each be utilized by someor all of the subsystems of the medical scan processing system, allowingsome or all of the subsystems and/or client devices to retrieve, edit,add, or delete entries to the one or more databases and/or files.

The one or more client devices 120 can each be associated with one ormore users of one or more subsystems of the medical scan processingsystem. Some or all of the client devices can be associated withhospitals or other medical institutions and/or associated with medicalprofessionals, employees, or other individual users for example, locatedat one or more of the medical institutions. Some of the client devices120 can correspond to one or more administrators of one or moresubsystems of the medical scan processing system, allowingadministrators to manage, supervise, or override functions of one ormore subsystems for which they are responsible.

Some or all of the subsystems 101 of the medical scan processing system100 can include a server that presents a website for operation via abrowser of client devices 120. Alternatively or in addition, each clientdevice can store application data corresponding to some or allsubsystems, for example, a subset of the subsystems that are relevant tothe user in a memory of the client device, and a processor of the clientdevice can display the interactive interface based on instructions inthe interface data stored in memory. For example, the website presentedby a subsystem can operate via the application. Some or all of thewebsites presented can correspond to multiple subsystems, for example,where the multiple subsystems share the server presenting the website.Furthermore, the network 150 can be configured for secure and/orauthenticated communications between the medical scan subsystems 101,the client devices 120 and the database storage system 140 to protectthe data stored in the database storage system and the data communicatedbetween the medical scan subsystems 101, the client devices 120 and thedatabase storage system 140 from unauthorized access.

FIG. 2A presents an embodiment of client device 120. Each client device120 can include one or more client processing devices 230, one or moreclient memory devices 240, one or more client input devices 250, one ormore client network interfaces 260 operable to more support one or morecommunication links via the network 150 indirectly and/or directly,and/or one or more client display devices 270, connected via bus 280.Client applications 202, 204, 206, 208, 210, 212, 214, and/or 216correspond to subsystems 102, 104, 106, 108, 110, 112, 114, and/or 116of the medical scan processing system respectfully. Each client device120 can receive the application data from the corresponding subsystemvia network 150 by utilizing network interface 260, for storage in theone or more memory devices 240. In various embodiments, some or allclient devices 120 can include a computing device associated with aradiologist, medical entity, or other user of one or more subsystems asdescribed herein.

The one or more processing devices 230 can display interactive interface275 on the one or more client display devices 270 in accordance with oneor more of the client applications 202, 204, 206, 208, 210, 212, 214,and/or 216, for example, where a different interactive interface 275 isdisplayed for some or all of the client applications in accordance withthe web site presented by the corresponding subsystem 102, 104, 106,108, 110, 112, 114 and/or 116. The user can provide input in response tomenu data or other prompts presented by the interactive interface viathe one or more client input devices 250, which can include amicrophone, mouse, keyboard, touchscreen of display device 270 itself orother touchscreen, and/or other device allowing the user to interactwith the interactive interface. The one or more processing devices 230can process the input data and/or send raw or processed input data tothe corresponding subsystem, and/or can receive and/or generate new datain response for presentation via the interactive interface 275accordingly, by utilizing network interface 260 to communicatebidirectionally with one or more subsystems and/or databases of themedical scan processing system via network 150.

FIG. 2B presents an embodiment of a subsystem 101, which can be utilizedin conjunction with subsystem 102, 104, 106, 108, 110, 112, 114 and/or116. Each subsystem 101 can include one or more subsystem processingdevices 235, one or more subsystem memory devices 245, and/or one ormore subsystem network interfaces 265, connected via bus 285. Thesubsystem memory devices 245 can store executable instructions that,when executed by the one or more subsystem processing devices 235,facilitate performance of operations by the subsystem 101, as describedfor each subsystem herein.

FIG. 3 presents an embodiment of the database storage system 140.Database storage system 140 can include at least one database processingdevice 330, at least one database memory device 340, and at least onedatabase network interface 360, operable to more support one or morecommunication links via the network 150 indirectly and/or directly, allconnected via bus 380. The database storage system 140 can store one ormore databases the at least one memory 340, which can include a medicalscan database 342 that includes a plurality medical scan entries 352, auser database 344 that includes a plurality of user profile entries 354,a medical scan analysis function database 346 that includes a pluralityof medical scan analysis function entries 356, an interface featuredatabase 348 can include a plurality of interface feature entries 358,and/or other databases that store data generated and/or utilized by thesubsystems 101. Some or all of the databases 342, 344, 346 and/or 348can consist of multiple databases, can be stored relationally ornon-relationally, and can include different types of entries anddifferent mappings than those described herein. A database entry caninclude an entry in a relational table or entry in a non-relationalstructure. Some or all of the data attributes of an entry 352, 354, 356,and/or 358 can refer to data included in the entry itself or that isotherwise mapped to an identifier included in the entry and can beretrieved from, added to, modified, or deleted from the database storagesystem 140 based on a given identifier of the entry. Some or all of thedatabases 342, 344, 346, and/or 348 can instead be stored locally by acorresponding subsystem, for example, if they are utilized by only onesubsystem.

The processing device 330 can facilitate read/write requests receivedfrom subsystems and/or client devices via the network 150 based onread/write permissions for each database stored in the at least onememory device 340. Different subsystems can be assigned differentread/write permissions for each database based on the functions of thesubsystem, and different client devices 120 can be assigned differentread/write permissions for each database. One or more client devices 120can correspond to one or more administrators of one or more of thedatabases stored by the database storage system, and databaseadministrator devices can manage one or more assigned databases,supervise assess and/or efficiency, edit permissions, or otherwiseoversee database processes based on input to the client device viainteractive interface 275.

FIG. 4A presents an embodiment of a medical scan entry 352, stored inmedical scan database 342, included in metadata of a medical scan,and/or otherwise associated with a medical scan. A medical scan caninclude imaging data corresponding to a CT scan, x-ray, MRI, PET scan,Ultrasound, EEG, mammogram, or other type of radiological scan ormedical scan taken of an anatomical region of a human body, animal,organism, or object and further can include metadata corresponding tothe imaging data. Medical scans can be awaiting review or can havealready been reviewed by one or more users or automatic processes andcan include tentative diagnosis data automatically generated by asubsystem, generated based on user input, and/or generated from anothersource. Some medical scans can include final, known diagnosis datagenerated by a subsystem and/or generated based on user input, and/orgenerated from another source, and can included in training sets used totrain processes used by one or more subsystems such as the medical scanimage analysis system 112 and/or the medical scan natural languageanalysis system 114.

Some medical scans can include one or more abnormalities, which can beidentified by a user or identified automatically. Abnormalities caninclude nodules, for example malignant nodules identified in a chest CTscan. Abnormalities can also include and/or be characterized by one ormore abnormality pattern categories such as such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, for exampleidentified in a chest x-ray. Abnormalities can also include any otherunknown, malignant or benign feature of a medical scan identified as notnormal. Some scans can contain zero abnormalities, and can be identifiedas normal scans. Some scans identified as normal scans can includeidentified abnormalities that are classified as benign, and include zeroabnormalities classified as either unknown or malignant. Scansidentified as normal scans may include abnormalities that were notdetected by one or more subsystems and/or by an originating entity.Thus, some scans may be improperly identified as normal. Similarly,scans identified to include at least one abnormality may include atleast one abnormality that was improperly detected as an abnormality byone or more subsystems and/or by an originating entity. Thus, some scansmay be improperly identified as containing abnormalities.

Each medical scan entry 352 can be identified by its own medical scanidentifier 353, and can include or otherwise map to scan image data 410,and metadata such as scan classifier data 420, patient history data 430,diagnosis data 440, annotation author data 450, confidence score data460, display parameter data 470, similar scan data 480, training setdata 490, and/or other data relating to the medical scan. Some or all ofthe data included in a medical scan entry 352 can be used to aid a userin generating or editing diagnosis data 440, for example, in conjunctionwith the medical scan assisted review system 102, the medical scanreport labeling system 104, and/or the medical scan annotator system106. Some or all of the data included in a medical scan entry 352 can beused to allow one or more subsystems 101, such as automated portions ofthe medical scan report labeling system 104 and/or the medical scandiagnosing system 108, to automatically generate and/or edit diagnosisdata 440 or other data the medical scan. Some or all of the dataincluded in a medical scan entry 352 can be used to train some or allmedical scan analysis functions of the medical scan analysis functiondatabase 346 such as one or more medical scan image analysis functions,one or more medical scan natural language analysis functions, one ormore medical scan similarity analysis functions, one or more medicalreport generator functions, and/or one or more medical report analysisfunctions, for example, in conjunction with the medical scan imageanalysis system 112, the medical scan natural language analysis system114, and/or the medical scan comparison system 116.

The medical scan entries 352 and the associated data as described hereincan also refer to data associated with a medical scan that is not storedby the medical scan database, for example, that is uploaded by a clientdevice for direct transmission to a subsystem, data generated by asubsystem and used as input to another subsystem or transmitted directlyto a client device, or other data associated with a medical scan that isreceived and or generated without being stored in the medical scandatabase 342. For example, some or all of the structure and dataattributes described with respect to a medical scan entry 352 can alsocorrespond to structure and/or data attribute of data objects or otherdata generated by and/or transmitted between subsystems and/or clientdevices that correspond to a medical scan. Herein, any of the dataattributes described with respect to a medical scan entry 352 can alsocorrespond to data extracted from a data object generated by a subsystemor client device or data otherwise received from a subsystem, clientdevice, or other source via network 150 that corresponds to a medicalscan.

The medical scan image data 410 can include one or more imagescorresponding to a medical scan. The medical scan image data 410 caninclude one or more image slices 412, for example, corresponding to asingle x-ray image, a plurality of cross-sectional, tomographic imagesof a scan such as a CT scan, or any plurality of images taken from thesame or different point at the same or different angles. The medicalscan image data 410 can also indicate an ordering of the one or moreimage slices 412. Herein, a “medical scan” can refer a full scan of anytype represented by medical scan image data 410. Herein, an “imageslice” can refer to one of a plurality of cross-sectional images of themedical scan image data 410, one of a plurality of images taken fromdifferent angles of the medical scan image data 410, and/or the singleimage of the medical scan image data 410 that includes only one image.Furthermore “plurality of image slices” can refer to all of the imagesof the associated medical scan, and refers to only a single image if themedical scan image data 410 includes only one image. Each image slice412 can include a plurality of pixel values 414 mapped to each pixel ofthe image slice. Each pixel value can correspond to a density value,such as a Hounsfield value or other measure of density. Pixel values canalso correspond to a grayscale value, a RGB (Red-Green-Blue) or othercolor value, or other data stored by each pixel of an image slice 412.

Scan classifier data 420 can indicate classifying data of the medicalscan. Scan classifier data can include scan type data 421, for example,indicating that the scan is a CT scan, x-ray, Mill, PET scan,Ultrasound, EEG, mammogram, or other type of scan. Scan classifier data420 can also include anatomical region data 422, indicating for example,the scan is a scan of the chest, head, right knee, or other anatomicalregion. Scan classifier data can also include originating entity data423, indicating the hospital where the scan was taken and/or a user thatuploaded the scan to the system. If the originating entity datacorresponds to a user of one or more subsystems 101, the originatingentity data can include a corresponding user profile identifier 355and/or include other data from the user profile entry 354 of the user.Scan classifier data 420 can include geographic region data 424,indicating a city, state, and/or country from which the scan originated,for example, based on the basic user data 510 retrieved from the userdatabase 344 based on the originating entity. Scan classifier data canalso include machine data 425, which can include machine identifierdata, machine model data, machine calibration data, and/or contrastagent data, for example based on imaging machine data 514 retrieved fromthe user database 344 based on the originating entity data 423. The scanclassifier data 420 can include scan date data 426 indicating when thescan was taken. The scan classifier data 420 can include scan prioritydata 427, which can indicate a priority score, ranking, number in aqueue, or other priority data with regard to triaging and/or review. Apriority score, ranking, or queue number of the scan priority data 427can be generated by automatically by a subsystem based on the scanpriority data 427, based on a severity of patient symptoms or otherindicators in the risk factor data 432, based on a prioritycorresponding to the originating entity, based on previously generateddiagnosis data 440 for the scan, and/or can be assigned by theoriginating entity and/or a user of the system.

The patient history data 430 can include patient identifier data 431which can include basic patient information such as name or anidentifier that may be anonymized to protect the confidentiality of thepatient, age, and/or gender. The patient identifier data 431 can alsomap to a patient entry in a separate patient database stored by thedatabase storage system, or stored elsewhere. The patient history datacan include patient risk factor data 432 which can include previousmedical history, family medical history, smoking and/or drug habits,pack years corresponding to tobacco use, environmental exposures,patient symptoms, etc. The patient history data 430 can also includelongitudinal data 433, which can identify one or more additional medicalscans corresponding to the patient, for example, retrieved based onpatient identifier data 431 or otherwise mapped to the patientidentifier data 431. Some or all additional medical scans can beincluded in the medical scan database, and can be identified based ontheir corresponding identifiers medical scan identifiers 353. Some orall additional medical scans can be received from a different source andcan otherwise be identified. Alternatively or in addition, thelongitudinal data can simply include some or all relevant scan entrydata of a medical scan entry 352 corresponding to the one or moreadditional medical scans. The additional medical scans can be the sametype of scan or different types of scans. Some or all of the additionalscans may correspond to past medical scans, and/or some or all of theadditional scans may correspond to future medical scans. Thelongitudinal data 433 can also include data received and/or determinedat a date after the scan such as final biopsy data, or some or all ofthe diagnosis data 440. The patient history data can also include alongitudinal quality score 434, which can be calculated automatically bya subsystem, for example, based on the number of additional medicalscans, based on how many of the additional scans in the file were takenbefore and/or after the scan based on the scan date data 426 of themedical scan and the additional medical scans, based on a date rangecorresponding to the earliest scan and corresponding to the latest scan,based on the scan types data 421 these scans, and/or based on whether ornot a biopsy or other final data is included. As used herein, a “high”longitudinal quality score refers to a scan having more favorablelongitudinal data than that with a “low” longitudinal quality score.

Diagnosis data 400 can include data that indicates an automateddiagnosis, a tentative diagnosis, and/or data that can otherwise be usedto support medical diagnosis, triage, medical evaluation and/or otherreview by a medical professional or other user. The diagnosis data 440of a medical scan can include a binary abnormality identifier 441indicating whether the scan is normal or includes at least oneabnormality. In some embodiments, the binary abnormality identifier 441can be determined by comparing some or all of confidence score data 460to a threshold, can be determined by comparing a probability value to athreshold, and/or can be determined by comparing another continuous ordiscrete value indicating a calculated likelihood that the scan containsone or more abnormalities to a threshold. In some embodiments,non-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the scan contains one or moreabnormalities, can be included in diagnosis data 440 in addition to, orinstead of, binary abnormality identifier 441. One or abnormalities canbe identified by the diagnosis data 440, and each identified abnormalitycan include its own set of abnormality annotation data 442.Alternatively, some or all of the diagnosis data 440 can indicate and/ordescribe multiple abnormalities, and thus will not be presented for eachabnormality in the abnormality annotation data 442. For example, thereport data 449 of the diagnosis data 440 can describe all identifiedabnormalities, and thus a single report can be included in thediagnosis.

FIG. 4B presents an embodiment of the abnormality annotation data 442.The abnormality annotation data 442 for each abnormality can includeabnormality location data 443, which can include an anatomical locationand/or a location specific to pixels, image slices, coordinates or otherlocation information identifying regions of the medical scan itself. Theabnormality annotation data 442 can include abnormality classificationdata 445 which can include binary, quantitative, and/or descriptive dataof the abnormality as a whole, or can correspond to one or moreabnormality classifier categories 444, which can include size, volume,pre-post contrast, doubling time, calcification, components, smoothness,spiculation, lobulation, sphericity, internal structure, texture, orother categories that can classify and/or otherwise characterize anabnormality. Abnormality classifier categories 444 can be assigned abinary value, indicating whether or not such a category is present. Forexample, this binary value can be determined by comparing some or all ofconfidence score data 460 to a threshold, can be determined by comparinga probability value to a threshold, and/or can be determined bycomparing another continuous or discrete value indicating a calculatedlikelihood that a corresponding abnormality classifier category 444 ispresent to a threshold, which can be the same or different threshold foreach abnormality classifier category 444. In some embodiments,abnormality classifier categories 444 can be assigned one or morenon-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the corresponding classifier category 444is present. For example, an abnormality that is a nodule can includeabnormality classification data 445 assigned a spiculation value of “1”to indicate that spiculation is present, and a lobulation value of “0”to indicate that lobulation is not present. Some abnormality classifiercategories 444 can be assigned one of a plurality of set options. Forexample, a nodule texture category can be assigned to one of “solid”,“part solid/mixed”, or “non-solid/ground glass opacity”. As anotherexample, a nodule calcification category can be assigned to one of“popcorn”, “laminated”, “solid”, “non-central”, “central”, “absent”, or“stippled”. As another example, a nodule sphericity category can beassigned to one of “linear”, “ovoid”, or “round”. As another example, anodule internal structure category can be assigned to one of “soft”,“fluid”, “fat”, or “air”. As another example, abnormality classificationdata 445 can categorize an abnormality of a chest x-ray as one of“Airway Abnormality (bronchiectasis or bronchiolitis)”, “AnatomicVariant”, “Atalectasis and/or scarring”, “Consolidation”, “DiaphragmaticAbnormality”, “Enlarged Cardiac Contour”, “Foreign Body (non-medical)”,“Hyperlucent Thorax”, “Mediastinal Abnormality”, “MusculoskeletalAbnormality”, “Nodule and/or mass”, “Pleural Effusion”, “PleuralLesion”, “Post surgical change”, “Reticular Opacities”, “Skinabnormality (mole, nipple shadow)”, “Support devices and medicaldevices”, “Abdominal abnormality not otherwise covered”, or “other”.Some abnormality classifier categories can include a hierarchy ofsubcategories. For example, “Pulmonary Vasculature” can be “Plethora” or“Oligaemia”, and “Pulmonary Vasculature” that identified as “Plethora”can be “Diffuse” or “Focal/Multifocal”. Some categories can have aninfinite number of possible values, for example, where “size” is anexact numerical measure of any value.

The abnormality classifier categories 444 can also include a malignancycategory, and the abnormality classification data 445 can include amalignancy rating such as a Lung-RADS score, a Fleischner score, and/orone or more calculated values that indicate malignancy level, malignancyseverity, and/or probability of malignancy. Alternatively or inaddition, the malignancy category can be assigned a value of “yes”,“no”, or “maybe”. The abnormality classifier categories 444 can alsoinclude abnormality pattern categories 446 such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, and the abnormalityclassification data 445 for each abnormality pattern category 446 canindicate whether or not each of the abnormality patterns is present.

The diagnosis data 440 as a whole and/or the abnormality annotation data442 for each abnormality, can include custom codes or datatypesidentifying the binary abnormality identifier 441, abnormality locationdata 443 and/or some or all of the abnormality classification data 445of one or more abnormality classifier categories 444. Alternatively orin addition some or all of the abnormality annotation data 442 for eachabnormality and/or other diagnosis data 440 can be presented in aDigital Imaging and Communications in Medicine (DICOM) format or otherstandardized image annotation format, and/or can be extracted intocustom datatypes based on abnormality annotation data originallypresented in DICOM format. Alternatively or in addition, the diagnosisdata 440 and/or the abnormality annotation data 442 for each abnormalitycan be presented as one or more medical codes 447 such as SNOMED codes,Current Procedure Technology (CPT) codes, ICD-9 codes, ICD-10 codes, orother standardized medical codes used to label or otherwise describemedical scans.

Alternatively or in addition, the diagnosis data 440 can include naturallanguage text data 448 annotating or otherwise describing the medicalscan as a whole, and/or the abnormality annotation data 442 can includenatural language text data 448 annotating or otherwise describing eachcorresponding abnormality. In some embodiments, some or all of thediagnosis data 440 is presented only as natural language text data 448.In some embodiments, some or all of the diagnosis data 440 isautomatically generated by one or more subsystems based on the naturallanguage text data 448, for example, without utilizing the medical scanimage data 410, for example, by utilizing one or more medical scannatural language analysis functions trained by the medical scan naturallanguage analysis system 114. Alternatively or in addition, someembodiments, some or all of the natural language text data 448 isgenerated automatically based on other diagnosis data 440 such asabnormality annotation data 442, for example, by utilizing a medicalscan natural language generating function trained by the medical scannatural language analysis system 114.

The diagnosis data can include report data 449 that includes at leastone medical report, which can be formatted to include some or all of themedical codes 447, some or all of the natural language text data 448,other diagnosis data 440, full or cropped images slices formatted basedon the display parameter data 470 and/or links thereto, full or croppedimages slices or other data based on similar scans of the similar scandata 480 and/or links thereto, full or cropped images or other databased on patient history data 430 such as longitudinal data 433 and/orlinks thereto, and/or other data or links to data describing the medicalscan and associated abnormalities. The diagnosis data 440 can alsoinclude finalized diagnosis data corresponding to future scans and/orfuture diagnosis for the patient, for example, biopsy data or otherlongitudinal data 433 determined subsequently after the scan. Themedical report of report data 449 can be formatted based on specifiedformatting parameters such as font, text size, header data, bulleting ornumbering type, margins, file type, preferences for including one ormore full or cropped image slices 412, preferences for including similarmedical scans, preferences for including additional medical scans, orother formatting to list natural language text data and/or image data,for example, based on preferences of a user indicated in the originatingentity data 423 or other responsible user in the corresponding reportformatting data 570.

Annotation author data 450 can be mapped to the diagnosis data for eachabnormality, and/or mapped to the scan as a whole. This can include oneor more annotation author identifiers 451, which can include one or moreuser profile identifiers 355 of a user of the system, such as anindividual medical professional, medical facility and/or medical entitythat uses the system. Annotation author data 450 can be used todetermine the usage data 520 of a user profile entry 354. Annotationauthor data 450 can also include one or more medical scan analysisfunction identifiers 357 or other function identifier indicating one ormore functions or other processes of a subsystem responsible forautomatically generating and/or assisting a user in generating some orall of the diagnosis data, for example an identifier of a particulartype and/or version of a medical scan image analysis functions that wasused by the medical scan diagnosing system 108 used to generate part orall of the diagnosis data 440 and/or an interface feature identifier359, indicating an one or more interface features presented to a user tofacilitate entry of and/or reviewing of the diagnosis data 440. Theannotation author data can also simply indicate, for one or moreportions of the diagnosis data 440, if this portion was generated by ahuman or automatically generated by a subsystem of the medical scanprocessing system.

In some embodiments, if a medical scan was reviewed by multipleentities, multiple, separate diagnosis data entries 440 can be includedin the medical scan entry 352, mapped to each diagnosis author in theannotation author data 450. This allows different versions of diagnosisdata 440 received from multiple entities. For example, annotation authordata of a particular medical scan could indicate that the annotationdata was written by a doctor at medical entity A, and the medical codedata was generated by user Y by utilizing the medical report labelingsystem 104, which was confirmed by expert user X. The annotation authordata of another medical scan could indicate that the medical code wasgenerated automatically by utilizing version 7 of the medical scan imageanalysis function relating to chest x-rays, and confirmed by expert userX. The annotation author data of another medical scan could indicatethat the location and a first malignancy rating were generatedautomatically by utilizing version 7 of the medical scan image analysisfunction relating to chest x-rays, and that a second malignancy ratingwas entered by user Z. In some embodiments, one of the multiplediagnosis entries can include consensus annotation data, for example,generated automatically by a subsystem such as the medical scanannotation system 106 based on the multiple diagnosis data 440, based onconfidence score data 460 of each of the multiple diagnosis data 440,and/or based on performance score data 530, 630, or 720 of correspondinguser, medical scan analysis function, or interface feature,respectfully, identified in the annotation author data for eachcorresponding one of the multiple diagnosis data 440.

Confidence score data 460 can be mapped to some or all of the diagnosisdata 440 for each abnormality, and/or for the scan as a whole. This caninclude an overall confidence score for the diagnosis, a confidencescore for the binary indicator of whether or not the scan was normal, aconfidence score for the location a detected abnormality, and/orconfidence scores for some or all of the abnormality classifier data.This may be generated automatically by a subsystem, for example, basedon the annotation author data and corresponding performance score of oneor more identified users and/or subsystem attributes such as interactiveinterface types or medical scan image analysis functions indicated bythe annotation author data. In the case where multiple diagnosis dataentries 440 are included from different sources, confidence score data460 can be computed for each entry and/or an overall confidence score,for example, corresponding to consensus diagnosis data, can be based oncalculated distance or other error and/or discrepancies between theentries, and/or can be weighted on the confidence score data 460 of eachentry. In various embodiments, the confidence score data 460 canincludes a truth flag 461 indicating the diagnosis data is considered as“known” or “truth”, for example, flagged based on user input, flaggedautomatically based on the author data, and/or flagged automaticallybased on the calculated confidence score of the confidence score dataexceeding a truth threshold. As used herein, a “high” confidence scorerefers to a greater degree or more favorable level of confidence than a“low” confidence score.

Display parameter data 470 can indicate parameters indicating an optimalor preferred display of the medical scan by an interactive interface 275and/or formatted report for each abnormality and/or for the scan as awhole. Some or all of the display parameter data can have separateentries for each abnormality, for example, generated automatically by asubsystem 101 based on the abnormality annotation data 442. Displayparameter data 470 can include interactive interface feature data 471,which can indicate one or more selected interface features associatedwith the display of abnormalities and/or display of the medical scan asa whole, and/or selected interface features associated with userinteraction with a medical scan, for example, based on categorizedinterface feature performance score data 720 and a category associatedwith the abnormality and/or with the medical scan itself. The displayparameter data can include a slice subset 472, which can indicate aselected subset of the plurality of image slices that includes a singleimage slice 412 or multiple image slices 412 of the medical scan imagedata 410 for display by a user interface. The display parameter data 470can include slice order data 473 that indicates a selected customordering and/or ranking for the slice subset 472, or for all of theslices 412 of the medical scan. The display parameter data 470 caninclude slice cropping data 474 corresponding to some or all of theslice subset 472, or all of the image slices 412 of the medical scan,and can indicating a selected custom cropped region of each image slice412 for display, or the same selected custom cropped region for theslice subset 472 or for all slices 412. The display parameter data caninclude density window data 475, which can indicate a selected customdensity window for display of the medical scan as a whole, a selectedcustom density window for the slices subset 472, and/or selected customdensity windows for each of the image slices 412 of the slice subset472, and/or for each image slice 412 of the medical scan. The densitywindow data 475 can indicate a selected upper density value cut off anda selected lower density value cut off, and/or can include a selecteddeterministic function to map each density value of a pixel to agrayscale value based on the preferred density window. The interactiveinterface feature data 471, slice subset 472, slice order data 473,slice cropping data 474, and/or the density window data 475 can beselected via user input and/or generated automatically by one or moresubsystems 101, for example, based on the abnormality annotation data442 and/or based on performance score data 720 of different interactiveinterface versions.

Similar scan data 480 can be mapped to each abnormality, or the scan asa whole, and can include similar scan identifier data 481 correspondingto one or more identified similar medical scans, for example,automatically identified by a subsystem 101, for example, by applyingthe similar scan identification step 1376 of the medical scan imageanalysis system 112 and/or applying medical scan similarity analysisfunction to some or all of the data stored in the medical scan entry ofthe medical scan, and/or to some or all corresponding data of othermedical scans in the medical scan database. The similar scan data 480can also correspond to medical scans received from another source. Thestored similarity data can be used to present similar cases to users ofthe system and/or can be used to train medical scan image analysisfunctions or medical scan similarity analysis functions.

Each identified similar medical scan can have its own medical scan entry352 in the medical scan database 342 with its own data, and the similarscan identifier data 481 can include the medical scan identifier 353each similar medical scan. Each identified similar medical scan can be ascan of the same scan type or different scan type than medical scan.

The similar scan data 480 can include a similarity score 482 for eachidentified similar scan, for example, generated based on some or all ofthe data of the medical scan entry 352 for medical scan and based onsome or all of the corresponding data of the medical scan entry 352 forthe identified similar medical scan. For example, the similarity score482 can be generated based on applying a medical scan similarityanalysis function to the medical image scan data of medical scans and402, to some or all of the abnormality annotation data of medical scansand 402, and/or to some or all of the patient history data 430 ofmedical scans and 402 such as risk factor data 432. As used herein, a“high” similarity score refers a higher level of similarity that a “low”similarity score.

The similar scan data 480 can include its own similar scan displayparameter data 483, which can be determined based on some or all of thedisplay parameter data 470 of the identified similar medical scan. Someor all of the similar scan display parameter data 483 can be generatedautomatically by a subsystem, for example, based on the displayparameter data 470 of the identified similar medical scan, based on theabnormality annotation data 442 of the medical scan itself and/or basedon display parameter data 470 of the medical scan itself. Thus, thesimilar scan display parameter data 483 can be the same or differentthan the display parameter data 470 mapped to the identified similarmedical scan and/or can be the same or different than the displayparameter data 470 of the medical scan itself. This can be utilized whendisplaying similar scans to a user via interactive interface 275 and/orcan be utilized when generating report data 449 that includes similarscans, for example, in conjunction with the medical scan assisted reviewsystem 102.

The similar scan data 480 can include similar scan abnormality data 484,which can indicate one of a plurality of abnormalities of the identifiedsimilar medical scan and its corresponding abnormality annotation data442. For example, the similarity scan abnormality data 484 can includean abnormality pair that indicates one a plurality abnormalities of themedical scan, and indicates one of a plurality of abnormalities of theidentified similar medical scan, for example, that was identified as thesimilar abnormality.

The similar scan data 480 can include similar scan filter data 485. Thesimilar scan filter data can be generated automatically by a subsystem,and can include a selected ordered or un-ordered subset of allidentified similar scans of the similar scan data 480, and/or a rankingof all identified similar scans. For example, the subset can be selectedand/or some or all identified similar scans can be ranked based on eachsimilarity score 482, and/or based on other factors such as based on alongitudinal quality score 434 of each identified similar medical scan.

The training set data 490 can indicate one or more training sets thatthe medical scan belongs to. For example, the training set data canindicate one or more training set identifiers 491 indicating one or moremedical scan analysis functions that utilized the medical scan in theirtraining set based on training set data 621, and/or indicating aparticular version identifier 641 of the one or more medical scananalysis functions that utilized the medical scan in their training set.The training set data 490 can also indicate which portions of themedical scan entry were utilized by the training set, for example, basedon model parameter data 623 of the corresponding medical scan analysisfunctions. For example, the training set data 490 can indicate that themedical scan image data 410 was included in the training set utilized totrain version X of the chest x-ray medical scan image analysis function,or that the natural language text data 448 of this medical scan was usedto train version Y of the natural language analysis function.

FIG. 5 presents an embodiment of a user profile entry 354, stored inuser database 344 or otherwise associated with a user. A user cancorrespond to a user of one or more of the subsystems such as aradiologist, doctor, medical professional, medical report labeler,administrator of one or more subsystems or databases, or other user thatuses one or more subsystems 101. A user can also correspond to a medicalentity such as a hospital, medical clinic, establishment that utilizesmedical scans, establishment that employs one or more of the medicalprofessionals described, an establishment associated with administeringone or more subsystems, or other entity. A user can also correspond to aparticular client device 120 or account that can be accessed one or moremedical professionals or other employees at the same or differentmedical entities. Each user profile entry can have a corresponding userprofile identifier 355.

A user profile entry 354 can include basic user data 510, which caninclude identifying information 511 corresponding to the user such as aname, contact information, account/login/password information,geographic location information such as geographic region data 424,and/or other basic information. Basic user data 510 can includeaffiliation data 512, which can list one or more medical entities orother establishments the user is affiliated with, for example, if theuser corresponds to a single person such as a medical professional, orif the user corresponds to a hospital in a network of hospitals. Theaffiliation data 512 can include one or more corresponding user profileidentifiers 355 and/or basic user data 510 if the correspondingaffiliated medical entity or other establishment has its own entry inthe user database. The user identifier data can include employee data513 listing one or more employees, such as medical professionals withtheir own user profile entries 354, for example, if the user correspondsto a medical entity or supervising medical professional of other medicalprofessional employees, and can list a user profile identifier 355and/or basic user data 510 for each employee. The basic user data 510can also include imaging machine data 514, which can include a list ofmachines affiliated with the user which can include machine identifiers,model information, calibration information, scan type information, orother data corresponding to each machine, for example, corresponding tothe machine data 425. The user profile entry can include client devicedata 515, which can include identifiers for one or more client devicesassociated with the user, for example, allowing subsystems 101 to senddata to a client device 120 corresponding to a selected user based onthe client device data and/or to determine a user that data was receivedby determining the client device from which the data was received.

The user profile entry can include usage data 520 which can includeidentifying information for a plurality of usages by the user inconjunction with using one or more subsystems 101. This can includeconsumption usage data 521, which can include a listing of, or aggregatedata associated with, usages of one or more subsystems by the user, forexample, where the user is utilizing the subsystem as a service. Forexample, the consumption usage data 521 can correspond to each instancewhere diagnosis data was sent to the user for medical scans provided tothe user in conjunction with the medical scan diagnosing system 108and/or the medical scan assisted review system 102. Some or all ofconsumption usage data 521 can include training usage data 522,corresponding to usage in conjunction with a certification program orother user training provided by one or more subsystems. The trainingusage data 522 can correspond to each instance where diagnosis feedbackdata was provided by user for a medical scan with known diagnosis data,but diagnosis feedback data is not utilized by a subsystem to generate,edit, and/or confirm diagnosis data 440 of the medical scan, as it isinstead utilized to train a user and/or determine performance data for auser.

Usage data 520 can include contribution usage data 523, which caninclude a listing of, or aggregate data associated with, usages of oneor more subsystems 101 by the user, for example, where the user isgenerating and/or otherwise providing data and/or feedback that can isutilized by the subsystems, for example, to generate, edit, and/orconfirm diagnosis data 440 and/or to otherwise populate, modify, orconfirm portions of the medical scan database or other subsystem data.For example, the contribution usage data 523 can correspond to diagnosisfeedback data received from user, used to generate, edit, and/or confirmdiagnosis data. The contribution usage data 523 can include interactiveinterface feature data 524 corresponding to the interactive interfacefeatures utilized with respect to the contribution.

The consumption usage data 521 and/or the contribution usage data 523can include medical scan entry 352 whose entries the user utilizedand/or contributed to, can indicate one or more specific attributes of amedical scan entry 352 that a user utilized and/or contributed to,and/or a log of the user input generated by a client device of the userin conjunction with the data usage. The contribution usage data 523 caninclude the diagnosis data that the user may have generated and/orreviewed, for example, indicated by, mapped to, and/or used to generatethe annotation author data 450 of corresponding medical scan entries352. Some usages may correspond to both consumption usage of theconsumption usage data 521 and contribution usage of the contributionusage data 523. The usage data 520 can also indicate one or moresubsystems 101 that correspond to each consumption and/or contribution.

The user profile entry can include performance score data 530. This caninclude one or more performance scores generated based on thecontribution usage data 523 and/or training usage data 522. Theperformance scores can include separate performance scores generated forevery contribution in the contribution usage data 523 and/or trainingusage data 522 and/or generated for every training consumption usagescorresponding to a training program. As used herein, a “high”performance score refers to a more favorable performance or rating thana “low” performance score.

The performance score data can include accuracy score data 531, whichcan be generated automatically by a subsystem for each contribution, forexample, based on comparing diagnosis data received from a user to datato known truth data such as medical scans with a truth flag 461, forexample, retrieved from the corresponding medical scan entry 352 and/orbased on other data corresponding to the medical scan, for example,received from an expert user that later reviewed the contribution usagedata of the user and/or generated automatically by a subsystem. Theaccuracy score data 531 can include an aggregate accuracy scoregenerated automatically by a subsystem, for example, based on theaccuracy data of multiple contributions by the user over time.

The performance data can also include efficiency score data 532generated automatically by a subsystem for each contribution based on anamount of time taken to complete a contribution, for example, from atime the request for a contribution was sent to the client device to atime that the contribution was received from the client device, based ontiming data received from the client device itself, and/or based onother factors. The efficiency score can include an aggregate efficiencyscore, which can be generated automatically by a subsystem based on theindividual efficiency scores over time and/or based on determining acontribution completion rate, for example based on determining how manycontributions were completed in a fixed time window.

Aggregate performance score data 533 can be generated automatically by asubsystem based on the aggregate efficiency and/or accuracy data. Theaggregate performance data can include categorized performance data 534,for example, corresponding to different scan types, different anatomicalregions, different subsystems, different interactive interface featuresand/or display parameters. The categorized performance data 534 can bedetermined automatically by a subsystem based on the scan type data 421and/or anatomical region data 422 of the medical scan associated witheach contribution, one or more subsystems 101 associated with eachcontribution, and/or interactive interface feature data 524 associatedwith each contribution. The aggregate performance data can also be basedon performance score data 530 of individual employees if the usercorresponds to a medical entity, for example, retrieved based on userprofile identifiers 355 included in the employee data 513. Theperformance score data can also include ranking data 535, which caninclude an overall ranking or categorized rankings, for example,generated automatically by a subsystem or the database itself based onthe aggregate performance data.

In some embodiments, aggregate data for each user can be further brokendown based on scores for distinct scan categories, for example, based onthe scan classifier data 420, for example, where a first aggregate datascore is generated for a user “A” based on scores from all knee x-rays,and a second aggregate data score is generated for user A based onscores from all chest CT scans. Aggregate data for each user can befurther based on scores for distinct diagnosis categories, where a firstaggregate data score is generated for user A based on scores from allnormal scans, and a second aggregate data score is generated for user Abased on scores from all scans that contain an abnormality. This can befurther broken down, where a first aggregate score is generated for userA based on all scores from scans that contain an abnormality of a firsttype and/or in a first anatomical location, and a second aggregate scoreis generated for A based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each user can be further based on affiliation data, where a rankingis generated for a medical professional “B” based on scores from allmedical professionals with the same affiliation data, and/or where aranking is generated for a hospital “C” based on scores for allhospitals, all hospitals in the same geographical region, etc. Aggregatedata for each user can be further based on scores for interfacefeatures, where a first aggregate data score is generated for user Abased on scores using a first interface feature, and a second aggregatedata score is generated for user A based on scores using a firstinterface feature.

The user profile entry can include qualification data 540. Thequalification data can include experience data 541 such as educationdata, professional practice data, number of years practicing, awardsreceived, etc. The qualification data 540 can also include certificationdata 542 corresponding to certifications earned based on contributionsto one or more subsystems, for example, assigned to users automaticallyby a subsystem based on the performance score data 530 and/or based on anumber of contributions in the contribution usage data 523 and/ortraining usage data 522. For example, the certifications can correspondto standard and/or recognized certifications to train medicalprofessionals and/or incentivize medical professionals to use thesystem. The qualification data 540 can include expert data 543. Theexpert data 543 can include a binary expert identifier, which can begenerated automatically by a subsystem based on experience data 541,certification data 542, and/or the performance score data 530, and canindicate whether the user is an expert user. The expert data 543 caninclude a plurality of categorized binary expert identifierscorresponding to a plurality of qualification categories correspondingto corresponding to scan types, anatomical regions, and/or theparticular subsystems. The categorized binary expert identifiers can begenerated automatically by a subsystem based on the categorizedperformance data 534 and/or the experience data 541. The categories beranked by performance score in each category to indicate particularspecialties. The expert data 543 can also include an expert ranking orcategorized expert ranking with respect to all experts in the system.

The user profile entry can include subscription data 550, which caninclude a selected one of a plurality of subscription options that theuser has subscribed to. For example, the subscription options cancorrespond to allowed usage of one or more subsystems, such as a numberof times a user can utilize a subsystem in a month, and/or to acertification program, for example paid for by a user to receivetraining to earn a subsystem certification of certification data 542.The subscription data can include subscription expiration information,and/or billing information. The subscription data can also includesubscription status data 551, which can for example indicate a number ofremaining usages of a system and/or available credit information. Forexample, the remaining number of usages can decrease and/or availablecredit can decrease in response to usages that utilize one or moresubsystems as a service, for example, indicated in the consumption usagedata 521 and/or training usage data 522. In some embodiments, theremaining number of usages can increase and/or available credit canincrease in response to usages that correspond to contributions, forexample, based on the contribution usage data 523. An increase in creditcan be variable, and can be based on a determined quality of eachcontribution, for example, based on the performance score data 530corresponding to the contribution where a higher performance scorecorresponds to a higher increase in credit, based on scan priority data427 of the medical scan where contributing to higher priority scanscorresponds to a higher increase in credit, or based on other factors.

The user profile entry 354 can include interface preference data 560.The interface preference data can include a preferred interactiveinterface feature set 561, which can include one or more interactiveinterface feature identifiers 359 and/or one or more interactiveinterface version identifiers 712 of interface feature entries 358and/or version identifiers of the interface features. Some or all of theinterface features of the preferred interactive interface feature set561 can correspond to display parameter data 470 of medical scans. Thepreferred interactive interface feature set 561 can include a singleinteractive feature identifier for one or more feature types and/orinterface types, and/or can include a single interactive interfaceversion identifier 712 for one or more interface categories. Thepreferred interactive interface feature set 561 can include a ranking ofmultiple features for the same feature type and/or interface type. Theranked and/or unranked preferred interactive interface feature set 561can be generated based on user input to an interactive interface of theclient device to select and/or rank some or all of the interfacefeatures and/or versions. Some or all of the features and/or versions ofthe preferred interactive feature set can be selected and/or rankedautomatically by a subsystem such as the medical scan interfaceevaluator system, for example based on interface feature performancescore data 720 and/or feature popularity data 721. Alternatively or inaddition, the performance score data 530 can be utilized by a subsystemto automatically determine the preferred interactive feature set, forexample, based on the scores in different feature-based categories ofthe categorized performance data 534.

The user profile entry 354 can include report formatting data 570, whichcan indicate report formatting preferences indicated by the user. Thiscan include font, text size, header data, bulleting or numbering type,margins, file type, preferences for including one or more full orcropped image slices 412, preferences for including similar medicalscans, preferences for including additional medical scans in reports, orother formatting preference to list natural language text data and/orimage data corresponding to each abnormality. Some or all of the reportformatting data 570 can be based on interface preference data 560. Thereport formatting data 570 can be used by one or more subsystems toautomatically generate report data 449 of medical scans based on thepreferences of the requesting user.

FIG. 6 presents an embodiment of a medical scan analysis function entry356, stored in medical scan analysis function database 346 or otherwiseassociated with one of a plurality of medical scan analysis functionstrained by and/or utilized by one or more subsystems 101. For example, amedical scan analysis function can include one or more medical scanimage analysis functions trained by the medical scan image analysissystem 112; one or more medical scan natural language analysis functionstrained by the medical scan natural language analysis system 114; one ormore medical scan similarity analysis function trained by the medicalscan image analysis system 112, the medical scan natural languageanalysis system 114, and/or the medical scan comparison system 116; oneor more medical report generator functions trained by the medical scannatural language analysis system 114 and/or the medical scan imageanalysis system 112, and/or the medical report analysis function trainedby the medical scan natural language analysis system 114. Some or all ofthe medical scan analysis functions can correspond to medical scaninference functions of the medical scan diagnosing system 108 or otherfunctions and/or processes described herein in conjunction with one ormore subsystems 101. Each medical scan analysis function entry 356 caninclude a medical scan analysis function identifier 357.

A medical scan analysis function entry 356 can include functionclassifier data 610. Function classifier data 610 can include input andoutput types corresponding to the function. For example the functionclassifier data can include input scan category 611 that indicates whichtypes of scans can be used as input to the medical scan analysisfunction. For example, input scan category 611 can indicate that amedical scan analysis function is for chest CT scans from a particularhospital or other medical entity. The input scan category 611 caninclude one or more categories included in scan classifier data 420. Invarious embodiments, the input scan category 611 corresponds to thetypes of medical scans that were used to train medical scan analysisfunction. Function classifier data 610 can also include output type data612 that characterizes the type of output that will be produced by thefunction, for example, indicating that a medical scan analysis functionis used to generate medical codes 447. The input scan category 611 canalso include information identifying which subsystems 101 areresponsible for running the medical scan analysis function.

A medical scan analysis function entry 356 can include trainingparameters 620. This can include training set data 621, which caninclude identifiers for the data used to train the medical scan analysisfunction, such as a set of medical scan identifiers 353 corresponding tothe medical scans used to train the medical scan analysis function, alist of medical scan reports and corresponding medical codes used totrain the medical scan analysis function, etc. Alternatively or inaddition to identifying particular scans of the training set, thetraining set data 621 can identify training set criteria, such asnecessary scan classifier data 420, necessary abnormality locations,classifiers, or other criteria corresponding to abnormality annotationdata 442, necessary confidence score data 460, for example, indicatingthat only medical scans with diagnosis data 440 assigned a truth flag461 or with confidence score data 460 otherwise comparing favorably to atraining set confidence score threshold are included, a number ofmedical scans to be included and proportion data corresponding todifferent criteria, or other criteria used to populate a training setwith data of medical scans. Training parameters 620 can include modeltype data 622 indicating one or more types of model, methods, and/ortraining functions used to determine the medical scan analysis functionby utilizing the training set 621. Training parameters 620 can includemodel parameter data 623 that can include a set of features of thetraining data selected to train the medical scan analysis function,determined values for weights corresponding to selected input and outputfeatures, determined values for model parameters corresponding to themodel itself, etc. The training parameter data can also include testingdata 624, which can identify a test set of medical scans or other dataused to test the medical scan analysis function. The test set can be asubset of training set 621, include completely separate data thantraining set 621, and/or overlap with training set 621. Alternatively orin addition, testing data 624 can include validation parameters such asa percentage of data that will be randomly or pseudo-randomly selectedfrom the training set for testing, parameters characterizing a crossvalidation process, or other information regarding testing. Trainingparameters 620 can also include training error data 625 that indicates atraining error associated with the medical scan analysis function, forexample, based on applying cross validation indicated in testing data624.

A medical scan analysis function entry 356 can include performance scoredata 630. Performance data can include model accuracy data 631, forexample, generated and/or updated based on the accuracy of the functionwhen performed on new data. For example, the model accuracy data 631 caninclude or be calculated based on the model error for determined forindividual uses, for example, generated by comparing the output of themedical scan analysis function to corresponding data generated by userinput to interactive interface 275 in conjunction with a subsystem 101and/or generated by comparing the output of the medical scan analysisfunction to medical scans with a truth flag 461. The model accuracy data631 can include aggregate model accuracy data computed based on modelerror of individual uses of the function over time. The performancescore data 630 can also include model efficiency data 632, which can begenerated based on how quickly the medical scan analysis functionperforms, how much memory is utilized by medical scan analysis function,or other efficiency data relating to the medical scan analysis function.Some or all of the performance score data 630 can be based on trainingerror data 625 or other accuracy and/or efficiency data determinedduring training and/or validation. As used herein, a “high” performancescore refers to a more favorable performance or rating than a “low”performance score.

A medical scan analysis function entry 356 can include version data 640.The version data can include a version identifier 641. The version datacan indicate one or more previous version identifiers 642, which can mapto version identifiers 641 stored in other medical scan analysisfunction entry 356 that correspond to previous versions of the function.Alternatively or in addition, the version data can indicate multipleversions of the same type based on function classifier data 610, canindicate the corresponding order and/or rank of the versions, and/or canindicate training parameters 620 associated with each version.

A medical scan analysis function entry 356 can include remediation data650. Remediation data 650 can include remediation instruction data 651which can indicate the steps in a remediation process indicating how amedical scan analysis function is taken out of commission and/orreverted to a previous version in the case that remediation isnecessary. The version data 640 can further include remediation criteriadata 652, which can include threshold data or other criteria used toautomatically determine when remediation is necessary. For example, theremediation criteria data 652 can indicate that remediation is necessaryat any time where the model accuracy data and/or the model efficiencydata compares unfavorably to an indicated model accuracy thresholdand/or indicated model efficiency threshold. The remediation data 650can also include recommissioning instruction data 653, identifyingrequired criteria for recommissioning a medical scan analysis functionand/or updating a medical scan analysis function. The remediation data650 can also include remediation history, indicating one or moreinstances that the medical scan analysis function was taken out ofcommission and/or was recommissioned.

FIG. 7 presents an embodiment of an interface features entries 358,stored in interface feature database 348 or otherwise associated withone of a plurality of interface features utilized by interactiveinterface 275 in conjunction with one or more subsystems. Eachinteractive feature entry 358 can have a corresponding featureidentifier 359. Some interactive feature entries can correspond to oneor a plurality of features that can be used by one or more interactiveinterface versions, for example, each mapped to a correspondinginteractive interface version identifier 712, where each interactiveinterface version can utilize multiple interface features. Someinteractive feature entries can correspond to multiple featurespresented in conjunction, for example, corresponding to an interactiveinterface version that presents a plurality of interface features. Someor all of the interactive feature entries can correspond to scan displayparameter data 470 of medical scan entries 352.

An interface features entry 358 can include interface feature type data710. This can indicate if the interface feature corresponds to aparticular type of textual and/or visual cue, a particular type of menuoption, a particular type of response input method, a particular type ofspatial interface layout, or other feature type. For example, theinterface feature type can correspond to an abnormality location displaycue. A first interface feature can correspond to a first abnormalitylocation display cue that includes circling an abnormality in red, asecond interface feature can correspond to a second abnormality locationdisplay cue that includes automatically displaying a cropped image slicethat includes the abnormality, and a third interface feature cancorrespond to a third abnormality location display cue that includesoverlaying an image slice with a confidence heat map based on theprobability that each location of the image slice corresponds to anabnormality. Multiple interface features of the same type can be scoredand or ranked to determine which particular feature is preferred for aparticular interactive interface version, for example, in conjunctionwith the medical scan interface feature evaluator system 110. An inputscan type corresponding to a scan type that can be utilized with theinterface feature can be indicated in an input scan category 711.

The interface feature type data 710 can further include one or moreinteractive interface version identifiers 712 corresponding to one ormore interface versions in which the feature is included. The versionidentifier can correspond to, for example, one or more subsystems 101that utilize the version when presenting interactive interface 275.Multiple interface features of different types can be included in aninteractive interface version, and each interactive interface versioncan be scored and or ranked to determine which particular version ispreferred, for example, in conjunction with the medical scan interfacefeature evaluator system 110. In some embodiments, each versioncorresponds to its own interface features entry 358, and can indicatethe set of interface features used by the interface version based ontheir corresponding interface feature identifiers 359. Thus, the one ormore interactive interface version identifiers 712 of an interactivefeature entry can correspond to interface feature identifiers 359 ofinterface versions stored in the database 348.

An interface features entry 358 can include performance score data 720that corresponds to how well users perform when interface feature isutilized by interactive interface 275. Performance score data 720 can bebased on aggregate accuracy score data 531 and/or efficiency score data532 of the user database 344, categorized by the correspondinginteractive interface feature data 524 indicated in the contributionusage data 523. The performance score data 720 can be generated by themedical scan interface feature evaluator system 110, and can be used topopulate some or all of the accuracy score data 531 and/or efficiencyscore data 532 of the user database 344 and the correspondinginteractive interface feature data 524 of the contribution usage data523. The interface feature performance score data 720 can be categorizedby data associated with each corresponding medical scan of usercontributions. For example, the interface feature performance score data720 can be categorized by scan classifier data 420, by abnormalitylocation, classifiers, or other data indicated in diagnosis data 440, byannotation author data 450, or by other data relevant to categorizingthe performance score data 720.

The interface feature performance score data can also include popularitydata 721 that corresponds to how many users select to use the interfacefeature and/or how many medical scans include display parameter dataautomatically generated by a subsystem corresponding to interfacefeature. For example, in embodiments where users can indicate their ownpreferred interface features in interface preference data 560, theinterface preference data 560 of the user database can be used todetermine a popularity score and/or ranking corresponding to theinterface feature.

The interface feature performance score data 720 can include rankingdata 722 which can include an overall ranking of interface feature,and/or ranking with respect to other interface features of the same typebased on accuracy, efficiency, and/or popularity. This can be used by asubsystem such as the medical scan interface feature evaluating system110 to populate the display parameter data 470 of some or all medicalscans in the medical scan database 342, for example, based on acategorized ranking of interface features corresponding to categoriesindicated by data of each the medical scan entry 352. As used herein, a“high” performance score refers to a more favorable performance orrating than a “low” performance score.

FIG. 8A presents an embodiment of the medical scan assisted reviewsystem 102. The medical scan assisted review system 102 can be used toaid medical professionals or other users in diagnosing, triaging,classifying, ranking, and/or otherwise reviewing medical scans bypresenting a medical scan for review by a user by transmitting medicalscan data of a selected medical scan and/or interface feature data ofselected interface features of to a client device 120 corresponding to auser of the medical scan assisted review system for display via adisplay device of the client device. The medical scan assisted reviewsystem 102 can generate scan review data 810 for a medical scan based onuser input to the interactive interface 275 displayed by the displaydevice in response to prompts to provide the scan review data 810, forexample, where the prompts correspond to one or more interface features.

In various embodiments, the medical scan assisted review system 102 isoperable to receive, via a network, a medical scan for review.Abnormality annotation data 442 is generated by identifying one or moreof abnormalities in the medical scan by utilizing a computer visionmodel that is trained on a plurality of training medical scans. Theabnormality annotation data 442 includes location data andclassification data for each of the plurality of abnormalities and/ordata that facilitates the visualization 825 of the abnormalities in thescan image data 410. Report data 830 including text describing each ofthe plurality of abnormalities is generated based on the abnormalitydata. The visualization 825 and the report data 830 (collectivelydisplayed annotation data 820) is transmitted to a client device. Adisplay device associated with the client device displays thevisualization 825 in conjunction with the medical scan via aninteractive interface, and the display device further displays thereport data 830 via the interactive interface.

The scan review data 810 can correspond to feedback such asconfirmation, additions, edits, or deletions of abnormality annotationdata 442, which can correspond to some or all of the diagnosis data 440,presented to the user in conjunction with the medical scan. The scanreview data 810 can also correspond to new annotation data or otherdiagnosis data entered by the user as part of a blind review and/orfirst review of the medical scan. The scan review data 810 can be usedto generate diagnosis data 440 for the medical scan. Alternatively or inaddition, the scan review data 810 can be compared to diagnosis data 440already associated with the medical scan, for example, to automaticallygenerate consensus data, to automatically generate performance scoredata 530 corresponding to the user, to automatically generateperformance score data 630 corresponding to the medical scan analysisfunction used to generate the diagnosis data 440, and/or toautomatically generate performance score data 720 corresponding to oneor more interface features presented by the interactive interface 275 ofthe medical scan assisted review system 102.

In this fashion, the medical scan assisted review system 102 can be usedin conjunction with other subsystems 101 of the medical scan processingsystem 100. For example, the medical scan assisted review system 102 canbe utilized to generate medical codes 447 in conjunction with the use ofthe of the medical scan report labeling system 104, to generateabnormality annotation data 442 and/or final consensus annotation datain conjunction with the use of the medical scan annotator system 106, toconfirm and/or edit diagnosis data 440 and to generate performance scoredata 630 based on expert review in conjunction with the use of themedical scan diagnosing system 108, and/or to generate performance scoredata 720 in conjunction with the user of the medical scan interfacefeature evaluator system 110.

Some or all of the displayed annotation data 820 presented inconjunction with the medical scan can be retrieved from the diagnosisdata 440 of the medical scan database 342 and/or be generatedautomatically by the medical scan assisted review system 102 in responseto selecting the medical scan for review, for example, by utilizing themedical scan image analysis system 112, the medical scan naturallanguage analysis system 114, and/or medical scan diagnosing system 108,or by utilizing functions stored locally by the medical scan assistedreview system 102. In some embodiments, the medical scan assisted reviewsystem automatically generates some or all of the displayed annotationdata 820 in response to receiving a medical scan via the network 150that was uploaded by the user or triaged from a different user orentity. In some embodiments, the medical scan assisted review systemautomatically generates some or all of the displayed annotation data 820after the medical scan is initially displayed to the user in response tothe user selecting a menu option or otherwise electing that thedisplayed annotation data 820 be generated automatically, and thedisplayed annotation data 820 can be displayed in conjunction with themedical scan once it is generated.

Features of the interactive interface presented to the user can includeone or more interface features and can be automatically determined orselected by the medical scan assisted review system 102, for example,based on the performance score data 720 of interface features in theinterface feature database 348 and/or generated by the medical scaninterface feature evaluating system 110, based on the display parameterdata 470 of the medical scan retrieved from the medical scan databaseand/or generated by the medical scan image analysis system 112, and/orbased on user preferences corresponding to the user of the client deviceretrieved from the user database 344. The user can also elect to turninterface features on or off, show or hide one or more interfacefeatures from view, and/or switch between multiple interface featuremodes at any time by selecting a menu option or based on other userinput to the client device, for example, where the medical scan assistedreview system can retrieve the features of the newly selected mode fromthe interface feature database 348 and/or where multiple interfaceversion types are stored in the application data.

The interactive interface 275 can present a medical scan to a user bydisplaying medical scan image data 410. This can include displaying asingle image slice 412 of the medical scan and/or displaying a selectednumber or automatically determined number of image slices 412simultaneously in adjacent views. If the medical scan only includes asingle image, the single image can be displayed. If multiple imageslices are available, the interactive interface 275 can allow a user toscroll through ordered slice images of the medical scan, where the imageslices are displayed one at a time sequentially in a forwards orbackwards direction based on user input to move a scroll bar displayedby the interface, user input to arrows presented on a keyboard of theclient device, user input to a scroll wheel on a mouse of the clientdevice, or other user input indicating the user elects to navigatesequentially forwards or backwards through the slice images. Theinteractive interface can allow a user to jump to particular slices ofthe medical scan, for example, where a slice is selected via user inputto the interactive interface based on a slice number, thumbnail image, apoint selected on a scroll bar, or other indicator. The interactive userinterface can also allow a user to zoom in on a selected region of aparticular slice, display multiple slices at the same time, or otherwisenavigate the medical scan data visually based on selecting menu optionspresented by the interactive user interface or based on other userinput.

As described herein, “jumping” to an image slice 412 selected by theuser or selected automatically by the medical scan assisted reviewsystem 102 can include immediately displaying the selected image slice412 instead of the currently displayed image slice 412. In otherembodiments, “jumping” to a selected image slice includes visuallyscrolling through previous image slices, for example, starting at thefirst image slice of the medical scan or currently displayed image sliceof the medical scan, visually scrolling forward or backward to theselected image slice. This effect can emulate the use of a scroll bar totransition from the current scroll through slices and/or provide theuser with context while jumping to the selected slice. Visuallyscrolling can include, for example, rapidly displaying, at apredetermined rate determined by the system or selected by the user, allof the intermediate image slices between the first or current slice andthe selected slice, or a uniformly distributed proportion ofintermediate slices between the first or current slice and the selectedslice, in forward sequence or backwards sequence based on the slicenumber ordering. For example, intermediate slices can be rapidlydisplayed in forward sequence until the selected slice is reached, or,if a current slice corresponds to a slice higher number than theselected slice, intermediate slices can be rapidly displayed in backwardorder until the selected slice is reached. In some embodiments, the rateat which the intermediate slices are rapidly displayed is constant. Inother embodiments, the rate at which intermediate slices are rapidlydisplayed is variable, for example, where the first intermediate slices,furthest away in the ordering from the selected slice, are displayed ata high rate, and where intermediate slices are automatically displayedat gradually lower rates as the currently displayed intermediate slicegets closer to the selected slice in the ordering. Displaying slices ata high rate can correspond to displaying each slice for a first timeduration that is shorter than a second time duration corresponding todisplaying slices at a low rate. In various embodiments, the user caninstead manually scroll to a selected slice using a scroll bar or othersimilar interface feature, and intermediate slices will be rapidlydisplayed in sequence in a similar fashion.

The interactive interface 275 can automatically present the displayedannotation data 820 to indicate abnormalities identified in diagnosisdata 440 of the medical scan by not only displaying text, but also bycircling or surrounding the abnormality by a shape, facilitatingautomatically jumping to a selected slice that includes the abnormality,outlining the abnormality, highlighting the abnormality or otherwiseindicating the abnormality in a contrasting and/or non-grayscale color,displaying an identified zoomed-in or cropped image slice that includesthe abnormality, displaying the image based on an identified Hounsfieldwindow selected to best display the abnormality, etc. Such means ofindicating the abnormality can correspond to one or more interfacefeatures indicated in the display parameter data 470 and/or interfacepreference data 560. Each abnormality can be indicated based onabnormality annotation data 442 such as the abnormality location data443, abnormality classification data 445, confidence score data 460,and/or other data included in the diagnosis data 440 or otherwiseassociated with the abnormality. If the diagnosis data 440 indicatesthat no abnormalities are present, the interactive interface can displaytext and/or a visual prompt indicating that no abnormalities weredetected in the medical scan.

The identified abnormalities can be indicated automatically uponpresenting the medical scan, for example, where a selected slice thatincludes the abnormality is initially presented with the abnormalitycircled in red. The user can indicate whether to display and/or hide theindicated abnormalities via a menu option or other user input to theinteractive interface. For example, a user may elect to first view themedical scan normally without detected abnormalities indicated to allowblind, unbiased review and/or diagnosis of the medical scan, and canlater elect to view one or more identified abnormalities after theblind, unbiased review. The identified abnormalities can be displayedvia on each image slice of the medical scan that include theabnormalities, via one or more interface features, as a user elects toscroll through image slices, jump to a selected image slice, or viewmultiple slices simultaneously. Multiple identified abnormalities of thesame medical scan can be indicated at the same time, for example, wheremultiple abnormalities are circled on a displayed image slice.Alternatively, multiple detected abnormalities can be displayed one at atime in sequence, and the user can indicate via a menu option or otheruser input to the interactive interface to progress to the next detectedabnormality. Progressing to the next detected abnormality can includeautomatically jumping to a new slice that includes the next abnormality,circling or otherwise indicating a new abnormality in a new location onthe same displayed image slice, changing the density window displayedbased on a new density window indicated in the display parameter data ofthe next abnormality, and/or otherwise changing the view based ondifferent display parameter data 470 associated with the next detectedabnormality.

The displayed annotation data 820 can include classification data in thereport data 830 of each abnormality, which can be displayed inconjunction with indicating each abnormality. For example, datadescribing size, volume, pre-post contract, doubling time,calcification, components, smoothness, texture, diagnosis data, one ormore medical codes, a malignancy rating such as a Lung-RADS score, orother classifying data. The classification data can be based onabnormality classification data 445 for one or more abnormalityclassifier categories 444, and/or can be based on other abnormalityannotation data 442. The classification data can be displayed accordingto one or more interface features indicated in the display parameterdata 470 and/or interface preference data 560. For example, theclassification data can be displayed as text or graphics which can beoverlaid on a displayed image slice and included with visualization 825,displayed report data 830 such as text displayed in a text windowadjacent to or on top of the displayed image slice, or otherwisedisplayed by the interactive interface.

Alternatively or in addition, some or all of the classification data isdisplayed via other visual means, for example, where quantitative dataof one or more classification categories is displayed in accordance witha color heat map. For example, the detected abnormality can be indicatedin accordance with a malignancy heat map, where different colors such asRGB values or other non-grayscale colors correspond to differentmalignancy scores, and where the abnormality is circled or surrounded bya shape, outlined, highlighted, or otherwise displayed in conjunctionwith the color corresponding to the malignancy score of the abnormalitybased on the malignancy heat map. In some embodiments, some or all ofthe abnormality classification data is displayed automatically inconjunction with indicating the abnormality. In other embodiments, someor all of the abnormality classification data is displayed in responseto selecting a menu option, clicking on or hovering over a region of theinteractive interface associated with the indicated abnormality, orotherwise electing to view the classification data via user input to theinteractive interface.

The displayed annotation data 820 can include confidence data of eachabnormality, which can be displayed in conjunction with indicating eachabnormality. For example, a confidence score corresponding to acalculated probability that the detected abnormality exists and/orcorresponding to a calculated probability that the detected abnormalityis malignant can be displayed as numerical text, which can be overlaidon a displayed image slice, displayed in a text window adjacent to thedisplayed image slice, or otherwise displayed according to one or moreinterface features indicated in the display parameter data 470 and/orinterface preference data 560.

The confidence data can correspond to confidence score data 460 and canbe retrieved from the medical scan database, and/or can be indicated inthe output of the medical scan analysis function used to automaticallygenerate the displayed annotation data 820 and/or can be generatedautomatically based on a performance score data 530 or qualificationdata 540 associated with a user or medical entity from which thedisplayed annotation data 820 originated. The confidence data can alsoinclude one or more classifier confidence scores corresponding to atleast one of the classifying categories, and the classifier confidencescore data can be displayed in conjunction with the classifier data, forexample, where a numerical score is displayed as text adjacent to thecorresponding text for each classifier category. Alternatively or inaddition, some or all of the confidence data is displayed via graphicsor other visual means, for example, where confidence score data of oneor more classification categories is displayed in accordance with aconfidence heat map, where different colors such as RGB values or othernon-grayscale colors correspond to different confidence scores, andwhere the abnormality is circled or surrounded by a shape, outlined,highlighted, or otherwise displayed in conjunction with the colorcorresponding to the confidence data of the abnormality based on theconfidence heat map.

In some embodiments, a raw probability heat map corresponding to rawpixel probability data can be displayed. For example, the probabilitymatrices generated by the medical scan image analysis system 112 thatinclude a plurality of abnormality probabilities mapped to a pluralityof pixel locations for each image slices of a medical scan can bereceived by the medical scan assisted review system 102. Every pixel, oruniformly distributed proportion of pixels, of one or more displayedimage slices 412 can be indicated in accordance with the raw probabilityheat map, where different colors such as RGB values or othernon-grayscale colors correspond to different raw abnormality probabilityscores. The density values corresponding to each pixel can be preservedin the display of each image slice 412 by determining a darkness orlightness of the pixel color based on each density value.

In embodiments where multiple heat maps are employed, for example, if amalignancy heat map is employed and a confidence score heat map isemployed, the user can elect to switch to a different heat map viewbased on user input, and/or the same image slice can be displayed inmultiple adjacent windows, where each window corresponds to a differentheat map. The one or more heat maps can be displayed for each imageslice in the medical scan as the user elects to scroll through imagesslices, jump to a selected image slice, or view multiple image slicessimultaneously. If multiple windows are employed to display the sameslice corresponding to different heat maps views, the user can use asingle scroll bar or other single user input indicator to jump to adifferent image slice, and the multiple windows can simultaneouslydisplay the same new indicated image slice in accordance with their heatmap view.

In some embodiments, the some or all of the confidence data is displayedautomatically in conjunction with indicating the abnormality. In otherembodiments, some or all of the confidence data is displayed in responseto selecting a menu option, clicking on or hovering over a region of theinteractive interface associated with the indicated abnormality, orotherwise electing to view the confidence data via user input to theinteractive interface. In some embodiments, the user can selectconfidence score thresholds for the confidence data via a sliding bar orother user input, and only abnormality annotation data and/orcategorized classification data with a corresponding confidence scorethat meets, exceeds, or otherwise compares favorably to thecorresponding confidence score threshold set by the user will bedisplayed. For example, if the user sets a calcification confidencescore threshold to 90%, and the confidence score corresponding to thecalcification data of the classification data is 85%, the calcificationdata will be hidden and/or removed from display by the interactiveinterface. The user may later elect to change the calcificationconfidence score threshold to 80%, and the calcification data will beadded to the display immediately and/or in response to later user inputindicating the user elects to view classification data.

Upon presenting each abnormality to the user for review, the interactiveinterface 275 presented by the medical scan assisted review system canpresent a menu option or otherwise prompt the user to provide scanreview data 810. The medical scan assisted review system 102 canautomatically generate scan review data 810 in response to user inputindicating the review of the displayed annotation data 820. The scanreview data 810 can include a binary value indicating whether or not theannotation is correct based on user input indicating that the indicatedabnormality is correct or incorrect, for example, where the user willindicate that the annotation is incorrect upon determining that the scanis normal, or determining the identified abnormality is not malignant.The scan review data 810 can also include a binary value indicatingwhether or not the abnormality location and/or each classificationcategory is correct or incorrect. The scan review data 810 can alsoinclude edited classification data based on user input modifying data ofone or more of the classification categories and/or edited location databased on user input modifying the location, for example, based on userinput identifying the actual location of the abnormality by circling,drawing a shape around, outlining, clicking on, zooming in on, cropping,or highlighting a new point or region that includes the abnormality,and/or moving borders outlining the identified region to change the sizeand or shape of the region, by erasing or otherwise removing highlightedor otherwise indicated portions of the region, by highlightingadditional portions adjacent to the region, by dragging a shapesurrounding the abnormality to a new location, or otherwise indicating anew location by utilizing one or one or more interface featuresindicated in the display parameter data 470 and/or interface preferencedata 560.

Generating the scan review data 810 can further include modifyingdiagnosis data 440, removing an identified abnormality from thediagnosis data 440, and/or adding a new abnormality and itscorresponding data to the diagnosis data 440. The scan review data 810can further be used by the system to generate performance score data 530corresponding to the user, to rank medical entities, to generate modelaccuracy data 631, to generate performance score data 720 correspondingto one or more interface features, to generate consensus data such as afinal consensus annotation and/or generate other data as described inconjunction with the other subsystems 101 described herein. In otherembodiments, another subsystem 101 can generate and/or modify some orall of this data immediately in response to determining the scan reviewdata 810 was generated and/or at a later time based on the user feedbackindicated in the scan review data 810.

In various embodiments, generating the scan review data 810 includesautomatically generating and/or updating one or more confidence scoresof the confidence data associated with the abnormality and/or one ormore classification category confidence scores associated with theabnormality, and confidence score data 460 of one or more abnormalitiesof the medical scan can be updated in the medical scan database 342accordingly. For example, when the user indicated that the abnormalityis correctly identified, the confidence score can be changed to 100%based on the abnormality being confirmed by user. The confidence scorecan also be assigned to a different value based on performance scoredata 530 and/or qualification data 540 of user. The confidence score canincrease slightly, for example, changing from 70% to 80% when theabnormality is confirmed by a novice user, and can increase moredrastically, for example, changing from 70% to 99%, when the abnormalityis confirmed by an expert user. In embodiments where the scan reviewdata 810 corresponds to annotations entered for the first time, aconfidence score can be automatically calculated based on the annotationand/or data in the user profile and mapped to the medical scan.

The confidence score for the edited abnormality data can also beassigned a confidence score of 100% based on determining the editedabnormality data of the scan review data 810, or assigned a newcalculated confidence score based on a calculated discrepancy betweenthe user edits and the identified confidence score, for example, wherethe confidence score is lowered from 70% to 66% if the discrepancy issmall, or lowered from 70% to 30% if the discrepancy is high. Forexample, calculating the discrepancy can include calculating theEuclidian distance between a feature vector corresponding to theoriginal abnormality data and the edited abnormality data, and thechange in confidence score can be proportional or otherwise a functionof the magnitude of the calculated discrepancy. The new confidence scorefor the edited diagnosis data can also be weighted based on theperformance score data 530 or qualification data 540 of the user, forexample, where the new confidence score is assigned a higher value whenthe user has a high performance score or qualification level and isassigned a lower value when the user has a low performance score orqualification level. Even if a discrepancy detected, the confidencescore can still increase based on the user having a high score, wherethe magnitude of the increase in the confidence score is proportionalto, or a function of, both the magnitude of the discrepancy and theperformance score of the user. New confidence scores can be assigned toeach classification category based on individual discrepancies in eachcategory and/or based on categorized performance data 534 or categorizedportions of qualification data 540. The discrepancies between the useredits of the scan review data 810 and the one or more identifiedabnormalities in the displayed annotation data 820, the feedbackcompletion time, or other data can further be used to generate and/orupdate performance score data 530 of user.

In various embodiments, the medical scan assisted review system 102 cangenerate annotation data in the scan review data 810 based on user inputfor new abnormalities identified in the medical scan, and/or to provideother descriptive information regarding the medical scan indicated bythe user in the scan review data. This annotation data can include DICOMformatted annotations of one or more identified abnormalities based onuser input. The user can elect to enter a new annotation mode based onselecting a menu option, can automatically enter the new annotation modeonce all identified abnormalities of the displayed annotation data 820are presented, and/or can automatically enter new annotation mode byclicking on or otherwise selecting a point or region of the displayedimage slice that does not include or is not within a threshold proximityto an already identified abnormality of the displayed annotation data820. Alternatively, the user may be presented with a medical scan thatneeds annotations provided for the first time, and the interactiveinterface 275 will automatically enter the new annotation mode describedin response to presenting the medical scan for annotation. The scanreview data 810 can thus include annotation data of new annotationsentered by the user in new annotation mode. Once generated, these newannotations created in the new annotation mode can be treated as a newentry to the identified abnormality data of the displayed annotationdata 820, and can be formatted to match format of the rest of theidentified abnormality data and can be displayed by the interactiveinterface 275 accordingly.

New annotation data of the scan review data 810 can be generated basedon the location of the one or more additional abnormalities indicated byuser input that can include circling, drawing a shape around, outlining,clicking on, highlighting, or otherwise identifying a point or regionthat includes the new abnormality, for example, based on the selectedinterface features. Classification data of the new annotation data canbe automatically generated by the medical scan assisted review system102 by applying an abnormality classifying function to the identifiedregion of the image slice or an identified region of a plurality ofimage slices, where the abnormality classifying function is included inthe medical scan analysis function database 346 and/or corresponds to amedical scan image analysis function generated by the medical scan imageanalysis system 112. This automatically generated classification datacan be included in the displayed annotation data and/or can be otherwisepresented to the user for review, and scan review data 810 correspondingto confirmation and/or edits of the automatically generatedclassification data can be generated based on user input in the samefashion as other review of displayed annotation data. Alternatively, allof the classification data for the new annotation data can be generatedin the scan review data based on user input describing and/orclassifying the newly identified abnormality. For example, the user caninteract with drop down menus, check boxes, and/or enter text directlyvia voice and/or keyboard input to enter the classification data. Theuser can be prompted to answer questions about the abnormality or thescan in its entirety. This newly entered classification data can includeabnormality classification data 445 for some or all of the abnormalityclassifier categories 444 and/or abnormality pattern categories 446described herein. In various embodiments, new annotation data can begenerated in accordance with some or all interface features presented inaccordance with FIGS. 10C-10V.

In various embodiments, the new abnormality can be identified by theuser on a single image slice, corresponding to the single image slicebeing viewed by the user. At least one adjacent image slice on eitherside can be identified automatically to include the abnormality, forexample, based on processing the identified abnormality or theidentified two-dimensional region to detect the abnormality in adjacentslices based a medical scan similarity analysis function such as medicalscan similarity analysis function or other function based on theidentified region in the image slice.

In various embodiments, the user can indicate when they are finishedidentifying new abnormalities by indicating that no furtherabnormalities are present via a menu option. The user can perform thisstep without identifying any new abnormalities, for example, in responseto determining that the scan is normal and that no abnormalities arepresent, or determining that the abnormalities already identified by themedical scan assisted review system 102 in the displayed annotation data820 are sufficient.

In some embodiments, the new abnormalities are entered by the user basedon a blind review of the medical scan. For example, the medical scanassisted review system 102 can hide the identified abnormalities of thedisplayed annotation data 820 or otherwise allow the user to blindlyidentify abnormalities in the scan without displaying annotation data.Thus, annotation data for these abnormalities are generated in the scanreview data 810 accordingly based on user input identifying theabnormalities blindly. Annotation data of the scan review data 810 canthen be compared to abnormality annotation data of the displayedannotation data 820 or other annotation data generated by a differentsource, for example, abnormality annotation data 442 assigned a truthflag 461. This comparison can be used to generate the scan review data810 and/or the confidence data, generate performance data for the user,to rank medical entities, to generate model accuracy data, to generateinterface evaluation data, to generate consensus data, and/or generateother data as described in conjunction with the other subsystemsdescribed herein. After receiving the user's annotation data, theinteractive interface can display the hidden abnormality data normallyand/or can visually indicate differences between the abnormalityindicated by the scan review data 810 and indicated in the displayedannotation data 820. For example, the interactive interface can displaythe user identified abnormality data indicated by scan review data 810in one color, and can display identified abnormalities of the displayedannotation data 820 generated by the different source in another color,can indicate regions of the user identified abnormalities that are oraren't overlapping regions of the identified abnormalities of thedisplayed annotation data 820, can indicate abnormalities of thedisplayed annotation data 820 that the user did not identify, canindicate abnormalities annotated by the user that are not included inthe displayed annotation data 820, or otherwise visually indicateautomatically detected differences between the scan review data 810 andthe displayed annotation data 820. Such means can be used to visuallycompare more than two sets of annotations received from more than twosets of sources, for example, in conjunction with annotations of a scanreceived from multiple users of the medical scan annotator system 106presented to an expert for final review.

A set of similar medical scans and/or set of full or cropped imageslices with similar abnormalities can be determined automatically, or inresponse to the user electing to view similar scans via user input, forsome or all identified abnormalities in the scan review data 810 and/orthe displayed annotation data 820, for example, by retrieving thesimilar scan data 480 of the medical scan from the medical scan dataand/or utilizing the medical scan similarity analysis function or one ormore portions of the similar scan identification step of the medicalscan image analysis system 112. This set of similar medical scans and/orset of full or cropped image slices can be presented to the user via theinteractive interface to aid the user in reviewing, confirming,populating, and/or editing the abnormality data, for example, based onsimilar scan display parameter data 483 or display parameter data 470associated with the similar scans. The identified set of similar medicalscans and/or set of full or cropped image slices for each abnormalitycan be further filtered to generate a smaller set of similar medicalscans and/or set of full or cropped image slices and/or to automaticallyselect a single medical scan and/or full or cropped image slice. The setcan be automatically filtered based on the medical scan similarityanalysis function, based on the similar scan filter data 485 associatedwith the similar scans, and/or can be automatically filtered based onother factors such by determining scans in the subset with a similarclassification and/or location, scans with a high longitudinal qualityscore 434, determining scans corresponding to patients with a similarpatient history and/or risk factors, ranking the identified set based onsimilarity score, or other criteria. This filtering criteria can bedetermined automatically, can be selected by the user via theinteractive interface, and/or user preference data that includes thefilter criteria can be retrieved from the user database such as similarscan filter data 485.

One or more adjacent windows can be presented that each display one ormore image slices of the full or filtered set of similar medical scans.Abnormality data for some or all of set of similar medical scans orand/or set of full or cropped image slices can be automaticallygenerated and/or retrieved from a database, and can be presented inconjunction with the display of the similar medical scans, for whereexample, where the abnormality is circled, highlighted or otherwiseindicated as discussed previously, and/or the classification data ispresented by the interactive interface as discussed previously. Invarious embodiments, only abnormalities of the similar scans that areidentified as similar abnormalities to one or more abnormalities of themedical scan, for example, based on the similar scan abnormality data484, will be identified and/or displayed by the medical scan assistedreview system 102. A single adjacent window can be designated fordisplaying similar scans, and the user can elect to switch betweendifferent similar scans based on user input. In various embodiments, thesimilar scans will be displayed to the user one at a time based on anordering designated in the similar scan filtering data 485, and the usercan elect to switch to the next similar scan in order when they havecompleted reviewing the current scan.

The user can elect to confirm and/or remove similar scans from the setbased on determining whether or not each similar scan is an appropriatematch, and this input can be indicated in similarity review data of thescan review data 810. The similar scan data 480 of the medical scanand/or a similarity score corresponding to the pair that includes themedical scan and similar scan can be modified based on the similarityreview data, for example, where the similar scan is removed from thesimilar scan data 480 in response to user input and/or where asimilarity score decreases based on user input indicating that thesimilar scan is an inappropriate match. This can also be used togenerate performance score data 630 for the medical scan similarityanalysis function, where the medical scan similarity analysis functionreceives a favorable score in response to the user confirming the matchand receives an unfavorable score in response to the user indicating thematch is not appropriate.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a similar scan will display image slices 412 corresponding to thesame image slice number in the sequence, for example, where the imageslice 10 out of 30 is displayed for the medical scan in the firstwindow, and image slice 10 out of 30 is displayed for the second medicalscan in the second window. In various embodiments, the displayed sliceof the similar scan can be selected by automatically determining theappropriate anatomical region corresponding to the currently displayedslice of the current scan. For example, the medical scan assisted reviewsystem can automatically select the appropriate corresponding imageslice in the previous scan based on the similar scan display parameterdata 483, by determining which image slice of the previous scan comparesmost favorably to the currently displayed image slice of the currentscan based on performing a medical scan similarity analysis function onthe image slices as described herein, by determining the outline of ananatomical region of the slice and determining a slice in the previousscan that most closely matches the anatomical region borders, and/or byanother visual comparison. In such embodiments, the user can use asingle scroll bar or other single user input indication to jump to adifferent image slice, and the multiple windows can simultaneouslydisplay the same numbered image slice or can scroll or jump by the samenumber of slices if different slice numbers are initially displayed. Insome embodiments, three or more adjacent windows corresponding to themedical scan and two or more similar scans are displayed, and can all becontrolled with the single scroll bar in a similar fashion.

In other embodiments, the selected slice of the similar medical scan inthe second window is a single or subset of slice images identified inthe set, is automatically selected based on identifying a slice image ofthe similar scan that includes a most similar view of the abnormality asthe currently displayed slice for the current scan. For example, slice12 of the medical image includes a view of the abnormality that is mostsimilar to slice 31 of the similar medical image, and slice 31 willautomatically be selected for display in conjunction with the userselecting to view slice 12. The user can then switch to view slice 13 ofthe medical image that includes a different view of the abnormality thatis most similar to slice 35 of the similar medical image, and slice 35will automatically be selected for display in conjunction with the userscrolling or jumping from slice 12 to slice 13 of the medical image. Inother embodiments, the user can scroll or otherwise navigate throughslices for the medical scan and one or more similar medical scansseparately.

In various embodiments, longitudinal data, such as one or moreadditional scans of longitudinal data 433 of the medical scan or ofsimilar scans, can be displayed in conjunction with the medical scanautomatically, or in response to the user electing to view longitudinaldata via user input. For example, the medical scan assisted reviewsystem can retrieve a previous scan or a future scan for the patientfrom a patient database or from the medical scan database automaticallyor in response to the user electing to view past patient data. One ormore previous scans can be displayed in one or more correspondingwindows adjacent to the current medical scan. For example, the user canselect a past scan from the longitudinal data for display. Alternativelyor in addition, the user can elect longitudinal parameters such asamount of time elapsed, scan type, electing to select the most recentand/or least recent scan, electing to select a future scan, electing toselect a scan at a date closest to the scan, or other criteria, and themedical scan assisted review system can automatically select a previousscan that compares most favorably to the longitudinal parameters. Theselected additional scan can be displayed in an adjacent windowalongside the current medical scan. In some embodiments, multipleadditional scans will be selected and can be displayed in multipleadjacent windows.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a selected additional scan will display image slices 412 determinedto corresponding with the currently displayed slice 412 of the medicalscan. As described with respect to selecting a slice of a selectedsimilar medical scan for display, this can be achieved based onselecting the image slice with a matching slice number, based onautomatically determining the image slice that most closely matches theanatomical region corresponding to the currently displayed slice of thecurrent scan, and/or based on determining the slice in the previous scanwith the most similar view of the abnormality as the currently displayedslice. The user can use a single scroll bar or other single user inputindication to jump to a different image slice, and the multiple windowscan simultaneously display the same numbered image slice, or can scrollor jump by the same number of slices if different slice numbers areinitially displayed. In some embodiments, three or more adjacent windowscorresponding to the medical scan and two or more additional scans aredisplayed, and can all be controlled with the single scroll bar in asimilar fashion.

The medical scan assisted review system 102 can automatically detectprevious states of the identified abnormalities based on the abnormalitydata, such as the abnormality location data. The detected previousstates of the identified abnormality can be circled, highlighted, orotherwise indicated in their corresponding window. The medical scanassisted review system 102 can retrieve classification data for theprevious state of the abnormality by retrieving abnormality annotationdata 442 of the similar abnormality mapped to the previous scan from themedical scan database 342. This data may not be assigned to the previousscan, and the medical scan assisted review system can automaticallydetermine classification or other diagnosis data for the previousmedical scan by utilizing the medical scan image analysis system asdiscussed. Alternatively or in addition, some or all of the abnormalityclassification data 445 or other diagnosis data 440 for the previousscan can be assigned values determined based on the abnormalityclassification data or other diagnosis data determined for the currentscan. Such abnormality classification data 445 or other diagnosis data440 determined for the previous scan can be mapped to the previous scan,and or mapped to the longitudinal data 433, in the database and/ortransmitted to a responsible entity via the network.

The medical assisted review system can automatically generate statechange data such as a change in size, volume, malignancy, or otherchanges to various classifiers of the abnormality. This can be achievedby automatically comparing image data of one or more previous scans andthe current scan and/or by comparing abnormality data of the previousscan to abnormality data of the current scan. In some embodiments, suchmetrics can be calculated by utilizing the medical scan similarityanalysis function, for example, where the output of the medical scansimilarity analysis function such as the similarity score indicatesdistance, error, or other measured discrepancy in one or moreabnormality classifier categories 444 and/or abnormality patterncategories 446. This calculated distance, error, or other measureddiscrepancy in each category can be used to quantify state change data,indicate a new classifier in one or more categories, to determine if acertain category has become more or less severe, or otherwise determinehow the abnormality has changed over time. In various embodiments, thisdata can be displayed in one window, for example, where an increase inabnormality size is indicated by overlaying or highlighting an outlineof the current abnormality over the corresponding image slice of theprevious abnormality, or vice versa. In various embodiments whereseveral past scans are available, such state change data can bedetermined over time, and statistical data showing growth rate changesover time or malignancy changes over time can be generated, for example,indicating if a growth rate is lessening or worsening over time. Imageslices corresponding to multiple past scans can be displayed insequence, for example, where a first scroll bar allows a user to scrollbetween image slice numbers, and a second scroll bar allows a user toscroll between the same image slice over time. In various embodimentsthe abnormality data, heat map data, or other interface features will bedisplayed in conjunction with the image slices of the past image data.

In various embodiments, longitudinal data 433 can also be retrieved foridentified similar medical scans. This can include one or more previousmedical scans taken before the identified similar medical scans and/orone or more future medical scans taken after the identified similarmedical scan. The previous and/or future scans can be displayed inadjacent windows for display, for example with automatic matching sliceselection and/or simultaneous scrolling as previously described. Invarious embodiments, some or all of the previous and/or future scans canbe the same type or different type of scan, for example where thecurrent scan and identified similar scan is an x-ray, and the futurescan of the identified scan is an MM. This can allow a user to viewricher data for a similar type of abnormality to aid in generatingand/or editing the diagnosis data. In various embodiments, for example,where the longitudinal data includes biopsy results or other diagnosisdata, the medical scan can automatically display these results inconjunction with the similar medical scans. In various embodiments,these results can be used by the user to generate, confirm, and/orreview the diagnosis data for the current medical scan. In variousembodiments, the medical scan assisted review system 102 or anothersubsystem 101 can automatically generate some or all of the scan reviewdata 810 and/or diagnosis data 440 for the current medical scan based onthe biopsy data and/or diagnosis data corresponding to automaticallyretrieved future medical scans that correspond to the automaticallyidentified the similar medical scans. This can also be used toautomatically generate or adjust confidence score data 460 with respectto the diagnosis data 440 of the medical scan, for example, where aconfidence score previously generated increases in response todetermining that diagnosis data 440 of a future medical scan of theidentified similar scan matches or otherwise compares favorably to thepreviously generated diagnosis data 440, and where a confidence scorepreviously generated decreases in response to determining that diagnosisdata 440 of a future medical scan of the identified similar scan matchesor otherwise compares unfavorably to the previously generated diagnosisdata 440.

The displayed annotation data 820 can include report data 830corresponding to each abnormality, or the entire scan, which can beautomatically generated and/or retrieved for display to the user inconjunction with each abnormality, allowing the user to confirm and/oredit the report data 830. The report data 830 can include some or all ofnatural language text data 448 and/or report data 449 of the medicalscan. Generating the report data 830 can include automaticallygenerating text describing each abnormality based on location dataand/or classifying data of the abnormality such as abnormalityannotation data 442, for example, by utilizing a medical scan reportgenerator function or other generator function trained on a training setof abnormality data or other diagnosis data and corresponding medicalreports, annotations, or other natural language data or text describingthe medical scan. Alternatively, some or all of the report data 830 cancorrespond to annotations and/or a report written by another user asnatural language text, for example retrieved from the medical scandatabase 342. The report data 830 can be generated and/or retrieved fordisplay by the medical scan assisted review system 102, in conjunctionwith initially displaying each abnormality and/or in response togenerating the scan review data 810 for each abnormality based on userfeedback. When report data 830 is presented to the user for review, theuser can elect to accept the report data 830 as it stands based on userinput to the interactive interface 275. The user can also elect torevise some or all of the report data 830, for example, where thenatural language text of the report is presented in a text editor windowand the user can type new text and/or delete existing text. Therevisions entered by the user can be included in the scan review data810, and can be used to modify diagnosis data 440 by extracting relevantterms from the revised natural language text, for example, by utilizingthe medical scan natural language analysis function. Alternatively, whenscan review data 810 is generated based on other user input indicatingedits to the displayed annotation data, for example, if a userhighlights a region with a new abnormality and/or modifies classifierdata of an existing category based on a menu option, modified naturaltext data can be automatically generated in response to the scan reviewdata 810 indicating that the displayed annotation data needs to beedited. Thus, original natural text data displayed in the report data830, for example, in conjunction with initially indicating anabnormality, can be automatically changed to the modified naturallanguage text in response to generating the scan review data 810, wherethe displayed report data 830 automatically reflects the modifiednatural language text.

The medical scan assisted review system 102 can generate a final reportin response to the user finalizing edits of the report data 830 and/orconfirming the report data 830. Report formatting data, for example,fixed criteria of the medical scan assisted review system 102 ingenerating reports, criteria stored in report formatting data 570 ofuser database 344 and/or criteria learned based on a training set offormatted reports can be used to determine font, text size, header data,bulleting or numbering type, margins, file type or other formatting tolist natural language text data corresponding to each abnormality togenerate the final report. Some or all of the report data 830 such as afinal formatted report, the raw natural language text data, or one ormore full or cropped images slices included in the report data 830 canbe displayed to the user via the interactive interface 275, mapped tothe medical scan in the medical scan database as report data 449, and/ortransmitted to a responsible entity based on triaging data.

The report data 830 also include one or more full or cropped imageslices of medical scans, such as full or cropped images for some or allof the identified abnormalities of the displayed annotation data 820,full or cropped images of the similar abnormalities selected from theidentified full or filtered set of similar scans, and/or full or croppedimages selected from one or more additional scans. For example,including the full medical scan or full similar scans in the report maynot be feasible, so one or more full or cropped image slices of themedical scan or similar scans can be included as embedded images in thereport. Full and/or cropped image slices of medical scan, identifiedadditional scans, and/or identified similar scans can be automaticallyselected based on the corresponding abnormality location data 443,display parameter data 470, similar scan display parameter data 483,scan review data 810 report formatting data 570, and/or other criteria.The interactive interface can also prompt the user to select one or moreimage slices to be included from the medical scan, identified additionalscans, and/or identified similar scans, and scan review data 810 canindicate the selected slices to be included in the report. The medicalscan assisted review system 102 can automatically select at least oneslice or region of the user selected slices of scan review data 810based on abnormality location data 443, display parameter data 470,similar scan display parameter data 483, report formatting data 570,and/or other criteria. Alternatively, the user can also indicatepreferred cropped regions for the report via user input, for example, byzooming in and/or selecting a cropped region of each image slice asdiscussed previously, and the scan review data 810 can indicate selectedcropped regions of each selected slice. Some or all of the cropped orfull user selected image slices in the scan review data 810 can be usedby the medical scan assisted review system 102 or another subsystem 101to automatically generate and/or modify display parameter data 470and/or similar scan display parameter data 483 for some or all of thecorresponding additional and/or similar medical scans and/or toautomatically determine and/or modify report formatting data 570 ofuser.

In some embodiments, the density data is lost when a report including afull or cropped slice image is generated, for example, when onlygrayscale image data is preserved. In such cases, generating each fullor cropped image for the report can include automatically determining anoptimal density window for each selected image slice, for example, basedon the current density window used to display the image slice, based ondisplay parameter data 470 and/or 483, based on report formatting data570, by automatically determining a density window that provides optimalcontrast the abnormality in the image, by automatically determining adensity window that provides optimal context for the abnormality in theimage, or otherwise optimally presents the abnormality in the selectedimage. The selected density window can be used to convert eachtwo-dimensional cropped or full image slice to a grayscale image in thereport.

In various embodiments, one or more abnormalities can be visuallyindicated some or all of the images included in the report data, wherean abnormality is circled, highlighted, or otherwise presented, forexample, by utilizing one or more interface features or other visualfeatures discussed in conjunction with the displayed annotation data820. In some embodiments, a heat map overlay is included in one or moreimages included in the report data 830. The presentation of abnormalitydata can be based on interface features or other data indicated in thedisplay parameter data 470, based on interface features or other dataindicated in report formatting data 570, and/or based on interfacefeatures or other data corresponding to a current view displayed by theuser interface. For example, in various embodiments, the user can selectan option presented by the interactive interface 275 to take one or morefull or cropped screenshot of the current view of one or more imageslices of additional and/or similar medical scans, presented with one ormore interface features that indicate one or more abnormalities, and thescreenshot can be included in the report data 830.

In addition to including full or cropped image slices of similarabnormalities, identifying information can also be included in thereport data 830 for each similar scan in the report. The identifyinginformation in the report data 830 can include some or all correspondingsimilar scan data 480, a medical scan identifier for similar scans,corresponding patient identifier, and/or hyperlink to access eachsimilar medical scan and/or patient data of each similar medical scan.Similar identifying information can also be included for some or alladditional medical scans included in the report data 830. In variousembodiments, a second user can access the report on their client devicecan click on or otherwise select a presented image of a similarabnormality, the medical scan identifier, patient identifier, and/or thehyperlink of an identified similar medical scan and/or identifiedadditional medical scan. The corresponding full medical scan image data410, corresponding diagnosis data 440, patient history data 430, orother data of the corresponding medical scan or 402 will automaticallybe retrieved from the medical scan database 342 or another storagesystem in response to this user input. This information can be displayedto the second user via an interactive interface displayed by a displaydevice corresponding to their client device 120. In various embodiments,the second user can interact with a displayed medical scan and/or 402based on interface features of interface preference data 560 of thesecond user and/or display parameter data 470 of the correspondingmedical scan and/or 402, for example, by interacting with interactiveinterface 275 presented on their client device 120 in conjunction withone or more features of the medical scan assisted review system 102.

In an example embodiment, the medical scan assisted review system 102can be used to review a medical scan selected by the user from thedatabase 342, automatically selected by a subsystem for transmission touser, or uploaded to the medical scan processing system 100 by user.Diagnosis data 440 can be retrieved from the database 342 and/or can beautomatically generated, for example, by utilizing a medical scan imageanalysis function selected by the medical scan diagnosing system 108. Inresponse to retrieving and/or generating the diagnosis data 440, themedical scan can automatically be presented to the user by utilizing themedical scan assisted review system 102, allowing a user to review thediagnosis data 440 presented as displayed annotation data 820. The scanreview data 810 can be generated automatically based on feedback enteredby the user as described herein, and report data 830 can be generatedautomatically based on confirmations and/or edits indicated in the scanreview data 810. The report data 830 and/or other modified informationindicated by the scan review data 810 can be mapped to the diagnosisdata 440 and/or report data 830. The usage data 520 and/or performancescore data for the user, performance score data 630 for the medical scanimage analysis function, and/or performance score data for one or moreinterface features utilized by the medical scan assisted review system102 can be generated and/or updated accordingly.

In some embodiments, this process or a similar process is executed bythe medical scan processing system in response to receiving a medicalscan from a client device 120, for example, immediately after a medicalscan is taken for a patient, the scan can automatically be uploaded tothe medical scan processing system 100 for processing to generate thediagnosis data 440, and will be immediately presented to the user ofclient device 120, or to another user selected by a subsystem 101, forexample, based on comparing scan classifier data 420 to performancescore data 530 and/or qualification data 540.

FIGS. 8B-8S present an example embodiment of the interactive interface275 presented in conjunction with the medical scan assisted reviewsystem 102, utilized as a lung screening assessment system. In variousembodiments, the lung screening assessment system is operable to receivea chest computed tomography (CT) scan that includes a plurality of crosssectional images. Nodule classification data of the chest CT scan isgenerated by utilizing a computer vision model that is trained on aplurality of training chest CT scans to identify a nodule in theplurality of cross sectional images and determine an assessment score. Alung screening report that includes the assessment score of the noduleclassification data is generated for display on a display deviceassociated with a user of the lung screening assessment system. Such alung screening assessment system can utilize some or all features of themedical scan assisted review system 102 and/or features of othersubsystems as described herein.

In FIG. 8B, the interactive interface 275 presents a listed queue ofscans for review. The insights column indicates how many nodules wereautomatically detected by the medical scan assisted review system. Theuser can select to view a particular lung screening scan by selectingthe corresponding row.

In FIG. 8C, the interactive interface 275 presents a particular scan inresponse to the user selecting the corresponding row in conjunction withthe view presented in FIG. 8B. The user can navigate through sliceimages, select regions of interest (ROIs), view annotations, and/or viewinsights generated automatically by the medical scan assisted reviewsystem 102. Here, the interactive interface 275 prompts the user toreview a finding, automatically identified by the medical scan assistedreview system 102.

In FIG. 8D, the interactive interface 275 automatically jumps to slice19 in response to user input electing to view the automatically detectedfinding, and interactive interface 275 automatically presents thevisualization 825 by circling the detected nodule and by listing thenodule details.

In FIG. 8E, the interactive interface 275 presents cropped slice imagesof medical scans corresponding to three similar nodules. Each of thethree similar nodules have corresponding longitudinal data, and areautomatically selected based on having at least three years of dataavailable, along with final biopsy results. The user can change thedisplayed view of any of the similar nodules by selecting from Y1 (year1 scan), Y2 (year 2 scan), Y3 (year 3 scan), or Dx (official biopsyresults). In FIG. 8F, the user elects to change the view of the croppedimage slice that includes the similar nodule corresponding to patient ID100999 from Y2 to Y1, and the interactive interface 275 automaticallypresents the Y1 cropped image that includes the nodule in response. InFIG. 8G, the user elects to view the final biopsy results, and finalbiopsy results corresponding to the similar nodule are displayed by theinteractive interface accordingly.

In FIG. 8H, the interactive interface 275 presents an option to view aprevious scan for the patient, and the interactive interface 275automatically displays the previous scan in an adjacent window. The usercan elect to scroll through the previous scan independently. In FIG. 8I,the interactive interface 275 automatically displays a correspondingimage slice by jumping to slice 19, and circles the previous state ofthe identified nodule in the previous scan. The user can scroll orotherwise navigate through both scans simultaneously as discussedherein.

In FIG. 8J, the interactive interface 275 presents automaticallygenerated report data 830 in a text window. Scan review data 810 can beautomatically generated in response to the user electing to edit some orall of the text in the report, can elect to approve the report as itstands, or elect to deny the report to remove the automaticallyidentified nodule entirely. In FIG. 8K, the interactive interfacedisplays that the first finding was approved in response to the scanreview data 810 indicating that the user elected to approve the firstreport, and prompts the user to select additional findings 2 or 3 forreview, or to view the final report. In FIG. 8L, the interactiveinterface indicates that the third finding was denied by the user inresponse to the scan review data 810 indicating that the user elected toapprove the third report, and this scan review data 810 is utilized bythe medical scan assisted review system 102 to learn from this userfeedback, for example, by generating performance score data 630 for amedical scan analysis function responsible for generating the diagnosisdata 440 associated with abnormality for the third finding.

In FIG. 8M, a final report is displayed to the user for review, andincludes the approved report data 830 for confirmed findings 1 and 2,but not denied finding 3. Scan review data 810 can be automaticallygenerated in response to the user electing to edit the final report,finalize the final report or add a new finding.

In FIG. 8N, the interactive interface enters the new abnormality mode inresponse to the scan review data 810 indicating that the user elects toenter a new finding. The user can type the report entry corresponding tothe new finding in a text window as shown in FIG. 8O. In FIG. 8P, theuser elects to select the region of interest corresponding to the newfinding, such as a nodule that was overlooked by the medical scanassisted review system, and the interactive interface 275 prompts theuser to indicate the region of interest by selecting points on the imageslice corresponding to vertices of a polygon that surrounds the nodule.In FIG. 8Q, the interactive interface displays five vertices 850selected by the user, and the interactive interface 275 prompts the userto indicate when they have finished by double clicking. The medical scanassisted review system 102 automatically generates scan review data 810corresponding to the new finding based on the text entered by the userin the text window and the polygon indicating the region of interest,automatically determined based on the five vertices 850 indicated by theuser, as presented in the interactive interface 275 of FIG. 8R inresponse to the user electing to approve the new finding, and this scanreview data 810 corresponding to the new finding can be added to themedical scan database and/or utilized in training sets used to improvethe performance of the medical scan assisted review system 102 or othersubsystems in subsequent uses.

FIG. 8S presents an example final report, automatically generated by themedical scan assisted review system 102 based on report data 830 inresponse to the user electing to finalize the final report. Naturallanguage text corresponding to the automatically identified findings 1and 2, as well as the report data 830 corresponding to the added findingindicated by the user are included. Cropped images of each finding inmedical scan are also included in the report automatically. A reportcompletion time and patient history are also included in the report.

FIGS. 8T-8Y present an example of a medical scan assisted review system102 utilized as a chest x-ray differential diagnosis system. In variousembodiments, the chest x-ray differential diagnosis system is operableto generate abnormality pattern data is generated for each of a receivedplurality of chest x-rays by identifying at least one pattern in eachchest x-ray corresponding to an abnormality by utilizing a computervision model that is trained on a plurality of training chest x-rays.Differential diagnosis data is generated for each chest x-ray based onthe abnormality pattern data. Filtering parameters are received from aclient device, and a filtered chest x-ray queue that includes a subsetof chest x-rays is selected based on the filtering parameters and thedifferential diagnosis data is generated for transmission to the clientdevice for display. Differential diagnosis data corresponding a chestx-ray indicated in chest x-ray selection data received from the clientdevice is transmitted to the client device for display via the displaydevice in conjunction with the chest x-ray.

In various embodiments, the differential diagnosis data for each of theplurality of chest x-rays includes a plurality of binary valuesindicating whether each of a plurality of abnormality pattern types arepresent or not present based on the abnormality pattern data. Forexample, the plurality of abnormality pattern types can includecardiomegaly, consolidation, effusion, emphysema, and/or fracture. Invarious embodiments, the abnormality pattern data includes confidencescore data corresponding to each of the plurality of abnormality patterntypes. Generating the differential diagnosis data includes comparing theconfidence score data for each of the plurality of abnormality patterntypes to a confidence score threshold. The binary values can indicatethe corresponding abnormality pattern type is present when thecorresponding confidence score data compares favorably to the firstconfidence score threshold, and can indicate the correspondingabnormality pattern type is not present when the correspondingconfidence score data compares unfavorably to the confidence scorethreshold. The confidence score threshold can be the same or differentfor each abnormality pattern type. The confidence score threshold can bedetermined automatically by the chest x-ray differential diagnosissystem or another subsystem, or can be set by a user via input to theinteractive interface.

A plurality of triaged and/or uploaded scans can be queued for review bya selected user, and the medical scan assisted review system 102 canpresent a listed queue of scans for review via another view of theinteractive interface 275 presented by the display device. This can bebased on a plurality of medical scans triaged to the user and/or inresponse to receiving a plurality of medical scans uploaded to themedical scan processing system 100 by user. The queue of scans can bedisplayed as line item data in a row, and corresponding data can bedisplayed such as include patient data, risk factor data, priority scoredata, a diagnosis summary, abnormality classifier data, confidence scoredata, or other data retrieved from the medical scan database 342 orgenerated by one or more subsystems 101.

The queue of scans can be displayed via the interactive interface 275 inan order based on an automatically generated priority score or apriority score retrieved from scan priority data 427 of the medical scandatabase 342. For example, the priority score can be based on a manuallyassigned score for an incoming scan, based on the date of the scan,based on a severity of patient symptoms, severity of previous diagnosisdata of the scan, or severity of the diagnosis data 440 automaticallygenerated for the medical scan, for example, based on a malignancy scoreindicated in diagnosis data 440. The user can re-sort and/or filter thequeue based on one or more selected criteria, selected via user input tothe interactive interface and/or determined based on queue criteriapreferences associated with the user, for example, mapped to userprofile entry 354. The selected criteria can include such as selectingto view the filtered list of scans based on criteria for one or moreselected abnormality pattern types. For example, the user can select toview the filtered list of scans where an abnormality patterncorresponding to cardiomegaly was detected, and can further select tosort the list of scans in reverse order by a confidence scorecorresponding to detection of cardiomegaly of confidence score data 460.As another example, consider the case where diagnosis data 440 hasmultiple entries corresponding to multiple diagnosis authors, forexample with a first entry was generated based on user input by anotheruser of the system and a second entry generated automatically byutilizing a medical scan image analysis function. The user can select toview the filtered list of scans where an abnormality was detected by thefirst diagnosis author that was not reported by the second diagnosisauthor, or vice versa. This can be used to quickly find discrepanciesbetween known diagnosis data and other diagnosis data generated by asubsystem, for example, by utilizing a medical scan analysis function.The user can continue to re-sort and/or further filter or un-filter thequeue by adding or removing sorting and/or filter criteria.

The user can be automatically presented a medical scan for review fromthe top of a sorted and/or filtered queue, or the original queue, by themedical scan assisted review system 102 as described herein.Alternatively or in addition, the user can select a scan from theoriginal queue of scans, or the re-sorted and/or filtered queue ofscans, based on clicking the corresponding row of the selected scan orother input to the interactive interface, and the selected medical scanwill be presented by the medical scan assisted review system 102 asdescribed herein. The medical scan review system can return to thedisplayed queue of scans after a user has completed review of thecurrent scan. The user can elect to re-order and/or re-filter the queueby providing new criteria, such as selecting a new confidence scorethreshold or selecting new abnormality pattern type. The user can alsoelect to select a new medical scan for review from the displayed queue.In other embodiments, medical scan review system can automaticallydisplay the next scan in the queue once the review of the current scanis complete, without returning the view of listed scans.

In some embodiments, the user can elect to confirm the diagnosis data440 for a selected medical scan without viewing the medical scan. Forexample, where the user can elect to confirm the diagnosis data 440 byselecting a menu option presented for each scan in the list of scans viathe interactive interface 275. This can be based on confidence scoredata 460 displayed as a line item in conjunction with the medical scan,for example, where the user elects to confirm diagnosis data 440 withoutviewing the scan because the displayed confidence score is 99%. The usercan select auto-confirm criteria, such as “automatically confirm allnormal scans” or “automatically confirm all diagnosis data thatindicates cardiomegaly with a confidence score that is greater than90%”. In such embodiments, the queue can be automatically filtered basedon the auto-confirm criteria, where automatically confirmed medicalscans are not listed. The auto-confirm criteria can be selected via userinput to the interactive interface 275 and/or determined based on queuecriteria preferences associated with the user, for example, mapped touser profile entry 354. In some embodiments, the auto-confirm criteriawill be learned and selected by a subsystem and/or the auto-confirmcriteria will be entered by an administrator.

In various embodiments, the medical scan assisted review system 102 willonly display medical scans with detected abnormalities, for example,where normal scans are included in the auto-confirm criteria. Forexample, medical scans automatically determined to be normal with atleast a threshold confidence score will not be presented to the user forreview, and will automatically be filtered from the queue. Diagnosisdata, report data, and/or a flag indicating the scan is normal willautomatically be mapped to the medical scan in the medical scan databaseand/or transmitted to the responsible medical entity without userintervention. For quality control, a threshold proportion of normalscans, or other auto-confirmed scans based on other criteria, can berandomly or pseudo-randomly selected and presented to the user forreview.

As illustrated in FIGS. 8T-8U, the interface can display progress ofprocessing a collection of chest x-ray models, for example, displayingthat 48 out of 1498 chest x-rays have been processed. Each processedchest x-ray can be shown in real-time, with patient information, thex-ray, and results. The results can be broken down into the differentabnormality pattern categories and can display if each of theabnormalities was found in previous findings, for example, in a humanreport, and also display if the finding was found by the medical scandifferential diagnosis system by utilizing a model such as a medicalscan image analysis function. An abnormality confidence threshold 895can be displayed, and an abnormality confidence score can be displayedfor each x-ray. For example, in FIG. 8T, the score is 74%, which isbelow the threshold 895. Thus, the results for the model show that noabnormality patterns were found. In FIG. 8U, the score is 98%, which isabove the threshold 895. The results show that the model foundconsolidation, effusion, and emphysema patterns, while the human reportonly found emphysema.

After all of the chest x-rays are processed, the interface can displaybatch statistics as shown in FIGS. 8V-8W. The user can choose filtercriteria by selecting to view one or more abnormality patterns that werefound or not found by a previous report or by the model. For example, inFIG. 8V, filtering criteria has been applied to display a queue of scanswhere cardiomegaly was not found by the medical scan differentialdiagnosis system. Conversely, in FIG. 8W, filtering criteria has beenapplied to display a queue of scans where cardiomegaly was found by themedical scan differential diagnosis system. FIG. 8X shows a view of thequeue of scans after the filtering criteria in FIG. 8W was applied, andthe interface can enable the user to scroll through the queue. Eachentry in the queue can display the identified abnormality pattern byhighlighting the pattern in red or otherwise indicating the pattern viaa visualization 825. The user can select one of the scans for furtherreview, as shown in FIG. 8Y, and can view the text of the originalreport.

FIGS. 9A and 9B present an embodiment of a medical scan report labelingsystem 104. The medical scan report labeling system 104 can be used toautomatically assign medical codes 447 to medical scans based on useridentified keywords, phrases, or other relevant medical condition termsof natural text data 910 in a medical scan report of the medical scan,identified by users of the medical scan report labeling system 104 viainteractive interface 275.

In various embodiments, the medical scan report labeling system 104 isoperable to transmit a medical report that includes natural languagetext to a first client device for display. Identified medical conditionterm data is received from the first client device in response. An aliasmapping pair in a medical label alias database is identified bydetermining that a medical condition term of the alias mapping paircompares favorably to the identified medical condition term data. Amedical code that corresponds to the alias mapping pair and a medicalscan that corresponds to the medical report are transmitted to a secondclient device of an expert user for display, and accuracy data isreceived from the second client device in response. The medical code ismapped to the first medical scan in the medical scan database when theaccuracy data indicates that the medical code compares favorably to themedical scan.

As illustrated in FIG. 9A, the medical scan report labeling system 104can retrieve the natural language text data 910 corresponding to amedical scan. For example, the medical scan report labeling system 104can utilize the medical scan database 342 of the database storage system140 to determine the natural language text data 910 of the medical scan,for example, by retrieving natural language text data 448, extractingnatural language text data from report data 449, and/or utilizing amedical report generating function to generate the natural language textdata 910 deterministically and automatically based on other data of themedical scan entry 352 such as medical scan image data 410, diagnosisdata 440, or other data. When the natural text data 910 isdeterministically and automatically generated, the natural language textdata 910 is a computer-generated representation in a form similar tonatural language text. The medical scan report labeling system 104 canutilize the user database 344 of the database storage system 140 toselect a user. Alternatively, a different medical scan database and/oruser database can be utilized by the medical scan report labeling system104. In some embodiments, the natural language text data 910 cancorrespond to a medical report written by a user of the system or othermedical professional and/or radiologist, for example, based on reviewingthe medical scan.

The subsystem memory device 245 can include a medical label aliasdatabase 920 that can include a plurality of alias mapping pairs 925that can be retrieved, edited, added, and/or removed by the medical scanreport labeling system 104. Each alias mapping pair 925 can include oneof a plurality of medical condition terms 922 and a corresponding one ofa plurality of medical codes 924, which can include SNOMED codes, CPTcodes, ICD-9 codes, ICD-10 codes, or other standardized medical codesused to label, annotate, or otherwise describe medical scans. Eachmedical condition term 922 in the medical label alias database 920 isunique, and each of the plurality of medical condition terms can includeat least one word. Thus, a corresponding medical code 924 is adeterministic function of a given medical condition term 922. Multiplemedical condition terms can map to the same medical code. The medicallabel alias database 920 can also be stored in the database storagesystem 140 and can be accessed by the medical scan report labelingsystem 104 and/or other subsystems 101 via network 150.

The medical scan report labeling system 104 can automatically select amedical report data 449 of a medical report from the database fortransmission to an automatically selected client device 120corresponding to a selected user of the system, such as user identifiedas a medical scan labeler in the qualification data 540 or other data ofa user profile entry 354. The selected client device 120 can display thenatural text data 910 of the medical report via the interactiveinterface 275 displayed by the display device 270 corresponding to theclient device 120. In some embodiments, only the natural text data 910is displayed, and medical scan image data 410 is not displayed to theuser. The interactive interface 275 can prompt the user to identify oneor more medical condition terms based on natural language text data 910.The client device 120 can generate identified medical condition termdata 930 based on the user input, for transmission to the medical scanreport labeling system 104. The identified medical condition term data930 can be generated based on a segment of consecutive words in thenatural language text data 910 of the medical report identified by theuser, based on a plurality of words in the natural language text data910 of the medical report identified by the user, where at least two ofthe plurality of words are not consecutive words in the natural languagetext data 910, and/or based on a plurality of words identified by theuser where at least one word in the plurality of words is not includedin the natural language text data of the medical report. For example,the interactive interface 275 can prompt the user to click on orhighlight words in the text. The interactive interface can prompt theuser to indicate which of a plurality of identified medical conditionterms each identified word or phrase is designated for by allowing theuser to highlight in multiple colors or to otherwise indicate a switchbetween medical condition terms. When the user identifies a relevantword or phrase of a medical condition term that is not explicitly statedin the natural language text data 910, the user can indicate this wordor phrase based on other user input such as text and/or voice input to akeyboard and/or microphone of the client device. The generatedidentified medical condition term data 930 can include the raw userinput, and/or can be pre-processed by the client device, for example,identifying separate groups of medical condition terms, parsing textand/or keyboard input, etc. The prompts and/or user input methodsutilized by the interactive interface to generate identified medicalcondition term data 930 can include one or more interface features, forexample, indicated in the interface preference data 560 or displayparameter data 470 corresponding to the medical scan, and/or the usercan interact with the user interface to identify medical terms based ontechniques described in conjunction with the medical scan assistedreview system 102.

Upon receiving the identified medical condition term data 930, themedical scan report labeling system 104 can automatically determine ifthe identified medical condition terms data 930 compares favorably toany medical condition terms 922 included in the medical label aliasdatabase 920. This can include searching for the exact same medicalcondition term 922 indicated by the identified medical condition termdata 930 in the database, determining a most similar medical conditionterm 922 term in the database that compares most favorably to theidentified medical condition term data 930, and/or identifying thematching medical condition term 932 in the database by determining oneor more keywords of the identified medical condition term data 930 thatmatch the one or more words of a medical condition term 922. Forexample, a medical condition term of a first alias mapping pair 925 anda string of words in the identified medical condition term data 930received from the client device 120 term data may differ by at least oneword, and determining that medical condition term 922 compares favorablyto the identified medical condition term data includes calculating asimilarity score between the medical condition term 922 and theidentified medical condition term data 930 and further includesdetermining that the similarity score compares favorably to a similaritythreshold, and/or further determining that the similarity score is morefavorable than similarity scores calculated for other medical conditionterms 922 and the identified medical condition term data 930. In someembodiments, determining the matching medical condition term 932includes performing a medical scan natural language analysis functionand/or a natural language similarity function trained by the medicalscan natural language analysis system 114 or the medical scan comparisonsystem 116. In various embodiments, the identified medical conditionterm data 930 can include multiple identified medical condition terms,and medical condition terms 922 in the database that compare favorablyto each of these multiple identified medical condition terms can beidentified separately.

If the medical scan report labeling system 104 determines that a medicalcondition term 922 in the database compares favorably to the identifiedmedical condition term data 930, the corresponding medical code 924indicated in the alias mapping pair of the medical condition term in thedatabase can be retrieved. This medical code 924 can be automaticallyassigned as the label for the medical scan, and can be mapped to themedical scan in the medical scan database accordingly as medical code447 and/or can be utilized to generate some or all of the diagnosis data440.

FIG. 9B illustrates an embodiment of the medical scan report labelingsystem 104 upon determining that no medical condition term 922 in thedatabase compare favorably to the identified medical condition term data930, the medical scan report can be transmitted to a client deviceassociated with an expert user or other medical professional. The expertmedical professional can be selected, for example, based on an overallor category-based and/or ranked performance score data 530 in the userdatabase 344, as described herein in accordance with other subsystemssuch as the medical scan annotator system 106. The expert user can alsobe identified based on a local database of identified experts of themedical scan report labeling system and/or based on the expert data 543of user profile entries 354 of the user database 344.

The natural language text data 910 of the medical report and/or thecorresponding medical scan image data 410 can be displayed to the expertuser via an interactive interface 275 displayed by a display devicecorresponding to the client device of the expert user, for example, inconjunction with the medical scan assisted review system 102 and/or byutilizing one or more interface features indicated by the displayparameter data 470 of the corresponding medical scan and/or interfacepreference data 560 of the expert user. The medical code can be enteredmanually by the expert user and/or can be generated automatically basedon scan review data 810 and/or other diagnosis feedback data provided bythe expert user to the interactive interface 275. The selected expertuser can enter one or more new alias mapping pair 925 based on themedical scan image data 410 and/or corresponding medical report naturallanguage text data 910, for example, by identifying one or moreappropriate medical codes 924 for the medical scan, and by determiningone or more words in the natural language text data that should map toeach of the one or more appropriate medical codes 924 in the future tocreate one or more corresponding medical condition terms 922. These oneor more new alias mapping pairs 925 can be added to the medical labelalias database 920. The one or more medical codes 924 identified by theexpert user can also be mapped to the medical scan in the medical scandatabase 342 as medical code 447 and/or can be utilized to generate someor all of the diagnosis data 440.

The expert may determine that the medical report otherwise does notinclude a new medical condition term that should map to the appropriatemedical code. The expert can still enter or indicate the appropriatemedical code via the interactive interface without indicatingcorresponding medical terms. The expert user can indicate via a menuoption or other user input to the interactive interface that the medicalreport is an inaccurate description of the medical image in response todetermining that the medical report does not include any medicalcondition term that appropriately maps to the medical code. Thisinformation can be used by the medical scan report labeling system 104or other subsystem to alert a medical professional and/or hospitalassociated with the medical report, for example, indicated in theoriginating entity data 423 or annotation author data 450 of the error,for example, by transmitting a notification to a client device 120 of acorresponding user. The medical scan report labeling system 104 cangenerate new performance score data for the medical professional and/ormedical entity, and the performance score data 530 of the user profileentry 354 corresponding to the medical professional and/or medicalentity can be automatically updated accordingly. A new medical reportcan be automatically generated by the medical scan report labelingsystem 104 or another subsystem and can be mapped to the medical scan,for example, by modifying the report data 449 and/or natural languagetext data 448 of the corresponding medical scan entry 352 in the medicalscan database. The new medical report can automatically be generatedbased on the medical code or other diagnosis feedback data provided bythe expert via the user interface, for example, utilizing scan reviewdata 810 to generate report data 830 as described in conjunction withthe medical scan assisted review system 102.

In some embodiments, even when the medical scan report labeling system104 retrieves a medical code of an alias mapping pair 925 based on amedical condition term in the database compares favorably to theidentified medical condition term data received from the client device,the medical code and corresponding medical scan are sent to a clientdevice of a selected expert user for display to the expert user via theinteractive interface displayed by a display device corresponding to theclient device of the expert user, allowing the expert user to verifythat the medical code is appropriate for the medical scan before it ismapped to the medical scan in the medical scan database. For example, arandom or pseudo-random sampling of proposed medical codes and theircorresponding medical scans can be sent to expert users automaticallybased on a fixed verification percentage. Different users can beassigned different verification percentages, for example, based onperformance score data 530 and/or qualification data 540 information inuser profile entries 354 of the user database 344. For example, a higherproportion of medical codes for a first user can be sampled than that ofa second user in response to the first user having less training, havinga lower performance score, being a newer user of the medical scan reportlabeling system, or another indication in the user profile entry 354requiring that the first user have more medical codes verified by anexpert. In various embodiments, different medical codes can be assigneddifferent verification percentages, for example, based on a severity ofthe medical code and/or a rarity of the medical code. In variousembodiments, different alias mapping pairs to the same medical code canbe assigned different verifications percentages, for example, based onan ambiguity or rarity of the medical condition term of the aliasmapping pair 925. Verification percentages can be assigned to some orall alias mapping pairs 925 in the medical alias mapping database, forexample, based on a corresponding ambiguity and/or rarity score, and canbe automatically generated and/or updated by the medical scan reportlabeling system based on aggregate accuracy data corresponding to usageof the alias mapping pair. Similarly, confidence score data 460 can bemapped to the medical codes 924 assigned by a normal user, or an expertuser, based on user experience such as performance score data 530 and/orqualification data 540 information and/or based on an ambiguity orrarity score of the corresponding medical code 924. Whether or not theassigned medical codes 924 are reviewed by an expert user can be basedon the confidence score data 460.

Upon presenting the proposed medical code 924 and medical scan to theuser via the interactive interface 275, the client device 120 cangenerate expert feedback data 960 based on user input by expert user inresponse to a prompt displayed by the display device via the interactiveinterface to provide feedback based on the medical scan and the medicalcode. For example, the expert user can indicate whether or not themedical code corresponds to the medical scan, and can provide acorrected medical code in response to determining the medical code isincorrect. The expert feedback data 960 can be included in the scanreview data 810 generated in conjunction with the medical scan assistedreview system 102, and/or can be otherwise generated based on the expertinput identifying abnormalities or otherwise annotating the medicalscan, where the expert feedback data 960 is generated based on a blindreview or can be based on response to displayed annotation data 820corresponding to the medical codes 924 determined based on the aliasmapping pair 925 identified by the user. The expert feedback data 960generated in response to expert user input can be transmitted to themedical scan report labeling system 104 or another subsystem 101, andthe medical scan database and/or the medical label alias database can beupdated accordingly.

In this fashion, the expert user can also indicate that the identifiedalias mapping pair is an inappropriate mapping based on the medicalscan, can indicate a modified alias mapping pair or indicate that thealias mapping pair be removed from the database, and/or can provide anew alias mapping pair that includes the corrected medical code and anappropriate medical condition term based on the medical report, forexample, if medical report is also sent to the client device of theexpert user and the natural text data of the medical report is alsodisplayed to the expert user via the interactive interface. Suchinformation can also be included in the expert feedback data 960generated by the client device.

The ambiguity score and/or rarity score of the identified alias mappingpair 925 can be automatically updated based on this expert feedback. Forexample, an ambiguity score can increase to indicate the identifiedalias mapping pair 925 is more ambiguous if the accuracy score dataindicates that the medical code was inappropriate and where theambiguity score decreases if the accuracy score data indicates that themedical code was appropriate. The rarity score can simply be updated toreflect that an identified alias mapping pair 925 is less rare each timethe identified alias mapping pair 925 is used and/or can be based on anumber of times the identified alias mapping pair 925 is used in arecent, fixed duration of time. The corresponding verificationpercentage can automatically be updated based on the updated ambiguityscore and/or rarity score.

In various embodiments, once a medical code 924 is mapped to a medicalscan in the medical scan database, a mapping that includes thecorresponding medical scan report and the medical code can be added to areport labeling training set that includes a subset of the plurality ofmedical reports and a corresponding set of medical codes. In variousembodiments, all medical reports are added to the report labelingtraining set once labeling is complete. In various embodiments, onlymedical reports corresponding to medical codes verified by an expert areadded to the training set. The medical scan report labeling systemand/or another subsystem such as the medical scan natural languageanalysis system 114 can generate a medical report analysis functionbased on the training set, for example, based on the natural languageprocessing techniques described in conjunction with medical scan naturallanguage analysis system 114. The medical report analysis function cantake a full medical report or natural language text data extracted froma medical report as input, and can generate one or more medical codes asoutput. The medical report analysis function be utilized toautomatically assign medical codes to other medical scans that have yetto be labeled and/or to verify previously assigned medical codes basedon a corresponding medical report. This can be utilized by one or moresubsystems to automatically assign a medical code 924 to a medical scanitself rather than sending the medical report to a user of the medicalscan report labeling system. This can also be used to automaticallyverify and/or correct medical codes generated based on user input to themedical scan report labeling system, for example, in addition to orinstead of expert review.

In some embodiments, the medical report analysis function can includeautomatically determining one or more keywords and/or a medicalcondition term in the natural language text 910 of the medical scanreport and can automatically search the medical label alias database fora match to determine the medical code. In various embodiments,generating and/or executing the medical report analysis function caninclude automatically generating new alias mapping pairs 925 to be addedto the database based on new identified medical condition terms 922,and/or can include automatically modifying or removing existing aliasmapping pairs from the database. Alternatively, the medical reportanalysis function can directly determine the medical code from thenatural language text data without utilizing the medical label aliasdatabase. A medical code 924 outputted by the medical report analysisfunction can be mapped to the medical scan directly in the medical scandatabase and/or can be sent to an expert user for verification based onthe conditions described previously, along with the medical scan and/orthe medical report.

Feedback from the expert via the interactive interface can be used togenerate model accuracy data, such as model accuracy data 631, generatedautomatically by the client device 120 of the expert user in response toa prompt displayed by the display device via the interactive interfaceto provide model accuracy data or other scan review data 810 based onthe medical code and the medical report and/or medical scan image data410, and the model accuracy data can be sent to the medical scan reportlabeling system or another subsystem 101. The model accuracy data can bemapped to the medical report analysis function, and/or existing modelaccuracy data can be automatically updated. This can include generatingand/or modifying model accuracy data 631 of the medical report analysisfunction in the medical scan analysis function database 346.

The medical code 924 generated by the medical report analysis functioncan be mapped to the medical scan in the medical scan database when themodel accuracy data indicates that the medical code is appropriate. Themedical report analysis function can be automatically modified based onthe model accuracy data indicating that the medical code is incorrectand/or based on a corrected medical code provided by the expert, forexample, in conjunction with the remediation process of the medical scandiagnosis system and/or based on the remediation data 650 of the medicalreport analysis function. The expert can also manually identifyparameters or rules of the medical report analysis function that needmodification based on such detected discrepancies, and these can be sentto the medical scan report labeling system with the model accuracy datafor integration into the model by the medical scan report labelingsystem 104 and/or the medical scan natural language analysis system 114.

In various embodiments, the medical report generating function can begenerated by the medical scan natural language analysis system 114 orother subsystem 101 based on a training set of determined medical codesgenerated by the medical scan report labeling system 104, and thecorresponding medical report, for example, where the medical reportgenerator function takes one or more medical codes assigned to a medicalscan as input, and automatically generates natural language text data448 and/or report data 449, based on natural language patterns and/orreport formatting patterns detected in conjunction with the medicalscans that are labeled by particular medical codes 924. The medical scannatural language analysis system 114 can also utilize the medical reportgenerating function, for example, determining medical condition termsthat are mapped to input medical code 924 to generate the naturallanguage text data 448 and/or report data 449.

In various embodiments, the medical scan image analysis function can begenerated by the medical scan image analysis system 112 or othersubsystem 101 based on a training set of determined medical codesgenerated by the medical scan report labeling system 104, and thecorresponding medical scan image data 410, for example, where themedical scan image analysis function takes medical scan image dataassigned to a medical scan as input, and automatically generates one ormore corresponding medical codes 924 as output, for example, withoututilizing natural language text data 448 and/or report data 449.

FIGS. 10A and 10B present an embodiment of a medical scan annotatorsystem 106. the medical scan annotator system 106 can be used to gatherannotations of medical scans based on review of the medical scan imagedata 410 by users of the system such as radiologists or other medicalprofessionals. Medical scans that require annotation, for example, thathave been triaged from a hospital or other triaging entity, can be sentto multiple users selected by the medical scan annotator system 106, andthe annotations received from the multiple medical professionals can beprocessed automatically by a processing system of the medical scanannotator system, allowing the medical scan annotator system toautomatically determine a consensus annotation of each medical scan.Furthermore, the users can be automatically scored by the medical scanannotator system based on how closely their annotation matches to theconsensus annotation or some other truth annotation, for example,corresponding to annotations of the medical scan assigned a truth flag461. Users can be assigned automatically to annotate subsequent incomingmedical scans based on their overall scores and/or based on categorizedscores that correspond to an identified category of the incoming medicalscan. In various embodiments, the medical scan annotator system canutilize the medical scan database 342 and/or user database 344, oranother other medical scan database and/or user database, for example,stored in local memory.

In various embodiments, the medical scan annotator system 106 isoperable to select a medical scan for transmission via a network to afirst client device and a second client device for display via aninteractive interface, and annotation data is received from the firstclient device and the second client device in response. Annotationsimilarity data is generated by comparing the first annotation data tothe second annotation data, and consensus annotation data is generatedbased on the first annotation data and the second annotation data inresponse to the annotation similarity data indicating that thedifference between the first annotation data and the second annotationdata compares favorably to an annotation discrepancy threshold. Theconsensus annotation data is mapped to the medical scan in a medicalscan database.

As illustrated in FIG. 10A, the medical scan annotator system 106 canselect a medical scan from the medical scan database 342 fortransmission via network 150 to one or more client devices 120associated with a selected user set 1010 corresponding to one or moreusers in the user database 344. A medical scan can be selected forannotation based on an assigned priority and/or based on a turn-basedqueue, for example, based on the scan priority data 427 of thecorresponding medical scan entry 352. The client device 120 of each userof the selected user set 1010 can display one or more received medicalscans to the via the interactive interface 275 displayed by a displaydevice corresponding to the client device 120, for example, bydisplaying medical scan image data 410 in conjunction with the medicalscan assisted review system 102.

The interactive interface 275 displayed by client devices 120 of eachuser in the selected user set 1010 can include a prompt to provideannotation data 1020 corresponding to the medical scan. This can includea prompt to provide a text and/or voice description via a keyboardand/or microphone associated with the client device. This can alsoinclude a prompt to indicate one or more abnormalities in the medicalscan, for example, by clicking on or outlining a region corresponding toeach abnormality via a mouse and/or touchscreen. For example, theinteractive interface can prompt the user whether or not an abnormalityis present. If the user indicates an abnormality is present, theinteractive interface can prompt the user to identify the region thatincludes the abnormality. This can include allowing the user to scrollthrough one or more slices, to identify one or more slices that containthe abnormality, and to select a region of the one or more slices thatcontains the abnormality. Once the region is identified, the interactiveinterface can prompt the user to provide descriptive informationclassifying an abnormality based on its size, type, etc. To aid the userin providing this information, the user interface can automatically cropone or more slices based on the identified region and/or zoom in on theidentified region. In various embodiments, the medical scan can bepresented for annotation by utilizing the medical scan assisted reviewsystem 102, for example, presented in the new annotation mode. Theinteractive interface 275 can present the medical scan by utilizinginterface features indicated in the display parameter data 470 and/orthe interface preference data 560 of the user, and/or the user canindicate the annotation data via the interactive interface 275 byutilizing interface features indicated in the display parameter data 470and/or the interface preference data 560 of the user. For example, someor all of the annotation data 1020 can correspond to, or beautomatically generated based on, the scan review data 810 generatedbased on the user input.

Annotation data 1020 can be transmitted from each client device of usersin the selected user set 1010 to the medical scan annotator system 106,for example, in response to receiving input data via the interactiveinterface indicating that the annotations are complete. The annotationdata 1020 can be raw annotation data corresponding directly to the userinput, or can be further processed by the client device beforetransmission. For example, a more precise region corresponding to eachabnormality can be determined automatically based on the user input andby determining actual boundary points of the abnormality by utilizingimage processing techniques and/or text and/or voice input can beprocessed and/or parsed, for example, by utilizing a medical scannatural language analysis function and/or medical report analysisfunction to generate medical codes 447 or other diagnosis data 440corresponding to the medical scan. Such processing can also be performedby the medical scan annotation system 106 and/or another subsystem whenthe raw annotation data is received.

The medical scan annotator system 106 can evaluate the set annotationdata 1020 received from the selected user set 1010 to determine if aconsensus is reached, and/or generate a final consensus annotation 1030,for example, by performing an annotation consensus function 1040. Forexample, consider a selected user set 1010 that includes three users. Iftwo users annotate a medical scan as “normal” and the third userannotates the medical scan as “contains abnormality”, the annotationconsensus function 1040 performed by medical scan annotator system 106may determine that the final consensus annotation 1030 is “normal” byfollowing a majority rules strategy. Alternatively, the medical scanannotator system 106 can determine that a consensus is not reachedbecause one of the users indicated that an abnormality is present, andthat the medical scan should not be passed off as normal because a levelof confidence that the scan is normal, determined by a calculatedconsensus confidence score 1050, does not exceed a consensus confidencethreshold. The confidence thresholds required for consensus can differfor different types of scans and/or severity of diagnosis.

If the medical scan annotator system 106 determines that a consensus isachieved, it can automatically generate the final consensus annotation1030, and can map this final consensus annotation to the medical imagein the medical scan database in diagnosis data 440, and/or transmit theconsensus annotation to an originating entity of the medical scan. Themedical scan annotator system 106 can also map the calculated consensusconfidence score to the medical image in the confidence score data 460.In some embodiments, a truth flag 461 will automatically be assigned toall final consensus annotation 1030 in the confidence score data 460and/or will automatically be assigned to final consensus annotation 1030that exceeds a truth threshold. In some embodiments, annotation data1020 received from each user and/or a corresponding annotationconfidence score can also be stored in the medical database, mapped tothe corresponding user and/or the corresponding performance score in theannotation author data 450.

In some embodiments, for example where annotation data 1020 includesseveral attributes, the annotation consensus function 1040 performed bythe medical scan annotation system 106 can determine whether a consensusis reached by calculating a difference between two or more receivedannotation data 1020, for example, by generating a feature vector forannotation data 1020 received from each user. Each feature vector can begenerated based on keywords, medical codes, abnormality location in themedical scan, abnormality size and/or shape in the medical scan, aclassification of the abnormality, or other attributes listed inannotation data 1020 received from each user. Performing the annotationconsensus function 1040 can further include calculating the Euclidiandistance or other vector distance between the two or more featurevectors. Performing the annotation consensus function 1040 can furtherinclude determining if consensus is reached by determining if theaverage of these Euclidian distances is below a certain discrepancythreshold, for example, after determining and removing outlierannotations from the set. Similarly, the annotation consensus function1040 can further include determining if consensus is reached by firstgenerating the final consensus annotation 1030, and then calculating theEuclidian distance between each annotation feature vector and the finalconsensus annotation 1030, where consensus is determined to reached andthe final consensus annotation is confirmed only if the average of thesecalculated Euclidian distances is below a certain discrepancy threshold.The annotation consensus function 1040 can calculate the final consensusannotation 1030 itself by creating a consensus feature vector, whereeach attribute of the consensus feature vector is determined bycalculating a mean, median or mode of each corresponding annotationfeature extracted from all of the received annotation data 1020. In thisfashion, calculating the consensus confidence score 1050 can includecalculating such an average Euclidian distance, where distances withlarger magnitudes correspond to lower or otherwise less favorableconsensus confidence scores 1050, and where distances with smallermagnitudes correspond to higher or otherwise more favorable consensusconfidence scores 1050. Alternatively or in addition, the finalconsensus annotation 1030 can be generated based on the most closelymatching annotations and/or based on another average, for example,calculating an average identified region that includes an abnormality.

The annotation consensus function 1040 further determine whether or notconsensus is reached based on overall or categorized performance scoredata 530 and/or qualification data 540 of each user in the selected userset 1010. For example, each annotation data 1020 can be weighted basedthe performance scores and/or qualifications of the corresponding user.In the example where two users annotate a medical scan as “normal” and athird user annotates a medical scan as “contains abnormality”, themedical scan annotator system 106 may determine that the consensus is“contains abnormality” based on the third user having a much higherperformance score and/or being more highly qualified than the first twousers. The final consensus annotation 1030 can be generated based on theannotation received from a user with the highest ranking in the categorycorresponding to the medical scan. The final consensus annotation 1030can be generated based on calculating a weighted average annotation bycomputing a weighted consensus feature vector, where feature vectors ofhigher ranked users receive a higher weight. In some embodiments, eachfeature of the feature vector can be computed using a different set ofuser weights, for example, where the different feature weights for eachuser is determined based on corresponding category-based performancescore data and/or qualification data.

Alternatively or in addition, the performance score data 720 associatedwith the interface features of the interactive interface 275 used byeach user to annotate the image can also be utilized to weight thedifferent annotations in reaching consensus. Such weights can be appliedwhen generating a consensus feature vector, where each annotationfeature vector is weighted according to the performance score data 720of one or more corresponding interface features used by thecorresponding user.

In some embodiments, confidence scores for each individual annotationcan also be calculated for each user's annotation, and the consensusconfidence score 1050 can be generated based on these confidence scores,for example, based on an average confidence score, based on confidencescores of annotation data that matches the final consensus annotation1030, etc. In some embodiments, the final consensus annotation 1030 canbe generated based on these confidence scores, for example, whereannotation feature vectors are weighted based on a correspondingconfidence score. The confidence scores for each annotation data 1020can be generated automatically, for example, based on performance scoredata 530 and/or performance score data 720 as discussed herein.Individual confidence scores and/or a consensus confidence score 1050can also be updated retroactively as new annotation data is received,for example, if new annotation data is received from another user, forexample corresponding to an expert review when consensus is not reached,and/or if new annotation data is automatically generated by a subsystemafter the consensus data is generated.

The medical scan annotator system 106 can also utilize auto-generatedannotation data of the medical scan to determine if consensus is reachedand/or to generate the final consensus annotation 1030. Theauto-generated annotation data can be automatically generated by medicalscan annotator system 106 by utilizing one or more medical scan analysisfunctions. The auto-generated annotation data can also be retrieved fromthe medical scan database 342 if it was generated by a subsystem 101previously. One or more auto-generated annotations can be assigned theirown weights and/or confidence scores, for example, based on the modelaccuracy data 631 and/or another determined performance of the functionand/or subsystem responsible for creating each auto-generatedannotation. Each auto-generated annotation data can be thus treated asan annotation from another user, and can be used to determine ifconsensus is reached and/or to generate the consensus annotation in thesame fashion.

Alternatively, the auto-generated annotation can be merely verifiedbased on the annotation data 1020 received from the selected user set1010 by determining that the user annotations are close enough to theauto-generated annotation based on the discrepancy threshold. Forexample, this process may be utilized by the medical scan diagnosingsystem 108 to perform the output quality assurance step 1107. Theauto-generated annotation can be sent to the selected user set 1010 aspart of this verification process, for example, displayed by eachinteractive interface 275 in conjunction with the medical scan assistedreview system 102 as displayed annotation data 820, and the annotationdata 1020 received from the selected user set 1010 can be includeverification of and/or corrections of the auto-generated annotation.Alternatively, the medical scan can be sent without the auto-generatedannotation and/or the auto-generated annotation can be hidden from viewas part of a blind review, to ensure that the users are not biased increating annotation data by the auto-generated annotation.

FIG. 10B illustrates an embodiment of the medical scan annotator system106 upon determining that a consensus is not achieved, for example,because the calculated consensus confidence score 1050 does not exceedthe consensus confidence threshold. The medical scan annotator systemcan select an expert user, for example, a user whose qualification data540 indicates they are an expert in the category corresponding to themedical scan or who otherwise is identified as an expert based on theirperformance score data. The expert can receive the medical scan on acorresponding client device and annotate the image, for example, wherethe interactive interface 275 displays the medical scan image data 410in conjunction with the medical scan assisted review system 102 andwhere the expert's annotations correspond to the scan review data 810,and where the interactive interface utilizes interface featuresindicated in the display parameter data 470 of the medical scan and/orindicated in the interface preference data 560 of the user profile entry354 of the expert user. The expert can view the annotation data 1020generated by the selected user set 1010, for example, presented as thedisplayed annotation data 820 of the medical scan assisted review system102. Annotation data 1020 of each user can be displayed one at a timeand the expert user can elect to advance to the next user's annotationdata 1020. Alternatively, all of the annotation data 1020 can bedisplayed simultaneously for example, in different colors correspondingto each user's annotations and/or overlaid as translucent, highlightedregions, for example, where a portion of the highlighted region is moreopaque when multiple users agree that the portion is included in theabnormality. In other embodiments, the annotation data 1020 can behidden from the expert user, and the expert user can enter their ownannotations in conjunction with a blind review to reduce bias.

Expert annotation data 1070 can be generated automatically, for exampleincluded in the scan review data 810, and can be transmittedautomatically to the medical scan annotation system 106. The medicalscan annotator system can automatically assign the received expertannotation data 1070 as the final consensus annotation 1030, and/or canassign a truth flag 461 to the expert annotation data 1070 in theconfidence score data 460 of the medical scan. Alternatively, the expertannotation data 1070 can be compared to the previous annotation data1020 and determine if consensus has been reached. For example, theexpert annotation data 1070 and the annotation data 1020 can becollectively utilized by the annotation consensus function 1040, wherethe expert annotation data 1070 is assigned its own, higher weight thanthe other annotations. If consensus has still not been reached, themedical scan annotation system can continue to transmit the image otherusers and processing received annotations until consensus is reached,for example, selecting a new selected user set 1010 and/or selecting anew expert user.

The user profile entries 354 of each user in the selected user set 1010and/or each expert user can be automatically updated by the medical scanannotator system 106 or another subsystem 101 by generating and/orupdating performance score data 530 for each user based comparing theirannotation to the final consensus annotation 1030. For example, theaccuracy score data 531 of the performance score data 530 can begenerated by calculating the Euclidian distance between a feature vectorof a user's annotation and the feature vector of the consensusannotation as described previously, where a higher performance score isassigned to a user whose annotation is a smaller Euclidian distance fromthe consensus, and a lower performance score is assigned to a user whoseannotation is a larger Euclidian distance from the consensus. Theefficiency score data 532 of the performance score data can beautomatically generated, for example, based on an annotation durationdetermined based on a difference between a first time that each userreceived the medical scan and a second time each user completed theannotation. The efficiency score data 532 can be further based on adifference between the annotation duration of each user and an averageannotation duration computed for annotation durations of the selecteduser set. Aggregate performance data for each user can be generateand/or updated based on past accuracy and/or efficiency scores, based onhow many scans have been annotated in total, based on measuredimprovement of the user over time, etc. Similarly, the performance scoredata 630 corresponding to medical scan analysis functions utilized togenerate the auto-generated annotation data can be generated and/orupdated by comparing the auto-generated annotation data to the finalconsensus annotation 1030 in a similar fashion and/or by comparing thecomputed annotation duration of a corresponding medical scan analysisfunctions to other computed annotation durations of other medical scananalysis functions that generated auto-generated annotation data for themedical scan.

The selected user set 1010 can be selected based on the performancescore data 530 and/or qualification data 540 of each user correspondingto previous uses only the medical scan annotation system 106, orcorresponding to usage of several subsystems 101. For example, a medicalprofessional with a user profile indicating that he/she ranks above acertain threshold in annotating CT scans and/or indicating that he/sheis highly qualified in the study of the lungs can be automaticallyselected by the medical scan annotator system to annotate a triagedmedical scan identified as a lug CT scan. The size of the selected userset 1010 that receive a medical scan can be optimized based on thequality of the users selected, for example, based on calculating theprobability of reaching consensus and/or calculating the probabilitythat a consensus confidence score will be above a confidence threshold,and ensuring the probability falls above a probability threshold. Forexample, a first medical scan can be sent to a two medical professionalswith high scores, qualifications, rankings, or correct annotationpercentages. A second medical scan may be sent to ten medicalprofessionals with lower scores or qualifications based on calculatingthat the probability of a correct consensus probability falls above aprobability threshold.

In some embodiments, the medical scan annotator system 106 can firstselect a medical scan for annotation automatically, and in response, theselected user set 1010 can be determined automatically to annotate theselected medical scan based on determining users with highly rankedoverall scores and/or based on categorized performance data 534 and/orqualification data 540 that corresponds to an identified scan classifierdata 420 of the selected medical scan. Alternatively or in addition, theselected user set 1010 can be determined based on the size of a queue ofmedical scans already assigned to each user. For example, the selecteduser set 1010 can correspond to users with matching qualifications thatcorrespond to the scan classifier data 420 and/or correspond to userswith the lowest queues of other medical scans to annotate.

In other embodiments, the medical scan annotator system 106 can firstdetermine one or more available users automatically, for example, basedon medical scan queue lengths for each user in the system and/or inresponse to one or more users requesting to annotate a medical scan. Insuch cases, some or all of these identified users can be added to theselected user set 1010, and the medical scan can be selected based oncorresponding categorized performance data 534, qualification data 540or other relevant user profile data of users in the selected user set1010.

FIGS. 10C-10V present example embodiments of a user interface of amedical scan annotator system 106, for example, presented in conjunctionwith the medical scan assisted review system 102. Some or all featurespresented in FIGS. 10C-10V can also be utilized in conjunction withother subsystems and can be included in the interface features. FIGS.10C-10G present interface features for chest CT nodule characterization,and can be displayed in conjunction with a chest CT scan, for example,as presented in FIGS. 8B-8S. Annotation data 1020 can be generated basedon user selections in the user interface, and can be used to populateabnormality classification data 445 for abnormality classifiercategories 444 such as “nodule spiculation”, “nodule lobulation”,“nodule texture”, “nodule calcification”, “nodule sphericity” and/or“nodule internal structure” for the associated medical scan. FIGS.10H-10J present interface features for presentation to a user inconjunction with an identifying chest CT nodule, allowing a user to addnew contours for one or more scans for a patient, for example, overmultiple years, and indicate malignancy. As shown in FIG. 10K, the scancan be presented in conjunction with these interface features. FIGS.10L-10O present interface features for presentation to a user inconjunction with identifying abnormalities in a chest x-ray. Users canclassify each abnormality and draw a shape around each abnormality inthe scan.

FIG. 10P presents a view of a chest x-ray presented via the interfacebefore a user identifies regions of interest, and FIG. 10Q presents aview of the chest x-ray via the interface after the user identifiesregions of interest of multiple abnormalities, indicated by sevenpolygons 1022. The user can generate polygons as described inconjunction with FIGS. 8P-8R. FIG. 10R presents interface features forcomparing chest x-ray severity for multiple patients, displayed inconjunction with multiple x-rays that can be displayed in adjacent viewsor can be displayed one at a time where the user can toggle betweenthem. A user can compare multiple scans corresponding to multiplepatients, and provide feedback indicating differences between thepatients, comparing if one patient's case is more severe than another,or determine which of two scans appears to be more normal.

FIGS. 10S-10V present interface features for chest x-ray triageclassification, displayed in conjunction with a chest x-ray. A user canselect abnormality classification data that can be used to generateannotation data 1020 and/or to populate abnormality classification data445. As shown, some or all abnormality classification categoriesdisplayed, which can be determined based on abnormality classifiercategories 444, can be presented, and hierarchal subcategories can bepresented in response to a user selecting one of a plurality ofabnormality classification categories that are present.

FIGS. 11A and 11B present an embodiment of a medical scan diagnosingsystem 108. A medical scan diagnosing system 108 can be used byhospitals, medical professionals, or other medical entities toautomatically produce inference data 1110 for given medical scans byutilizing computer vision techniques and/or natural language processingtechniques. This automatically generated inference data 1110 can be usedto generate and/or update diagnosis data 440 or other corresponding dataof the corresponding medical scan entry 352. The medical scan diagnosingsystem can utilize the medical scan database 342, user database 344,and/or medical scan analysis function database 346 by communicating withthe database storage system 140 via the network 150, and/or can utilizeanother medical scan database, user database, and/or function databasestored in local memory.

In various embodiments, the medical scan diagnosing system 108 isoperable to receive a medical scan. Diagnosis data of the medical scanis generated by performing a medical scan inference function on themedical scan. The first medical scan is transmitted to a first clientdevice associated with a user of the medical scan diagnosing system inresponse to the diagnosis data indicating that the medical scancorresponds to a non-normal diagnosis. The medical scan is displayed tothe user via an interactive interface displayed by a display devicecorresponding to the first client device. Review data is received fromthe first client device, where the review data is generated by the firstclient device in response to a prompt via the interactive interface.Updated diagnosis data is generated based on the review data. Theupdated diagnosis data is transmitted to a second client deviceassociated with a requesting entity.

The medical scan diagnosing system 108 can generate inference data 1110for medical scans by utilizing a set of medical scan inference functions1105, stored and run locally, stored and run by another subsystem 101,and/or stored in the medical scan analysis function database 346, wherethe function and/or parameters of the function can be retrieved from thedatabase by the medical scan diagnosing system. For example, the set ofmedical scan inference function 1105 can include some or all medicalscan analysis functions described herein or other functions thatgenerate inference data 1110 based on some or all data corresponding toa medical scan such as some or all data of a medical scan entry 352.Each medical scan inference function 1105 in the set can correspond to ascan category 1120, and can be trained on a set of medical scans thatcompare favorably to the scan category 1120. For example, each inferencefunction can be trained on a set of medical scans of the one or moresame scan classifier data 420, such as the same and/or similar scantypes, same and/or similar anatomical regions locations, same and/orsimilar machine models, same and/or similar machine calibration, sameand/or similar contrasting agent used, same and/or similar originatingentity, same and/or similar geographical region, and/or otherclassifiers. Thus, the scan categories 1120 can correspond to one ormore of a scan type, scan anatomical region data, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data 420. For example, a first medical scan inferencefunction can be directed to characterizing knee x-rays, and a secondmedical scan inference function can be directed to chest CT scans. Asanother example, a first medical scan inference function can be directedto characterizing CT scans from a first hospital, and a second medicalscan image analysis function can be directed to characterizing CT scansfrom a second hospital.

Training on these categorized sets separately can ensure each medicalscan inference function 1105 is calibrated according to its scancategory 1120, for example, allowing different inference functions to becalibrated on type specific, anatomical region specific, hospitalspecific, machine model specific, and/or region specific tendenciesand/or discrepancies. Some or all of the medical scan inferencefunctions 1105 can be trained by the medical scan image analysis systemand/or the medical scan natural language processing system, and/or somemedical scan inference functions 1105 can utilize both image analysisand natural language analysis techniques to generate inference data1110. For example, some or all of the inference functions can utilizeimage analysis of the medical scan image data 410 and/or naturallanguage data extracted from abnormality annotation data 442 and/orreport data 449 as input, and generate diagnosis data 440 such asmedical codes 447 as output. Each medical scan inference function canutilize the same or different learning models to train on the same ordifferent features of the medical scan data, with the same or differentmodel parameters, for example indicated in the model type data 622 andmodel parameter data 623. Model type and/or parameters can be selectedfor a particular medical scan inference function based on particularcharacteristics of the one or more corresponding scan categories 1120,and some or all of the indicated in the model type data 622 and modelparameter data 623 can be selected automatically by a subsystem duringthe training process based on the particular learned and/or otherwisedetermined characteristics of the one or more corresponding scancategories 1120.

As shown in FIG. 11A, the medical scan diagnosing system 108 canautomatically select a medical scan for processing in response toreceiving it from a medical entity via the network. Alternatively, themedical scan diagnosing system 108 can automatically retrieve a medicalscan from the medical scan database that is selected based on a requestreceived from a user for a particular scan and/or based on a queue ofscans automatically ordered by the medical scan diagnosing system 108 oranother subsystem based on scan priority data 427.

Once a medical scan to be processed is determined, the medical scandiagnosing system 108 can automatically select an inference function1105 based on a determined scan category 1120 of the selected medicalscan and based on corresponding inference function scan categories. Thescan category 1120 of a scan can be determined based one some or all ofthe scan classifier data 420 and/or based on other metadata associatedwith the scan. This can include determining which one of the pluralityof medical scan inference functions 1105 matches or otherwise comparesfavorably to the scan category 1120, for example, by comparing the scancategory 1120 to the input scan category of the function classifier data610.

Alternatively or in addition, the medical scan diagnosing system 108 canautomatically determine which medical scan inference function 1105 isutilized based on an output preference that corresponding to a desiredtype of inference data 1110 that is outputted by an inference function1105. The output preference designated by a user of the medical scandiagnosing system 108 and/or based on the function of a subsystem 101utilizing the medical scan diagnosing system 108. For example, the setof inference functions 1105 can include inference functions that areutilized to indicate whether or not a medical scan is normal, toautomatically identify at least one abnormality in the scan, toautomatically characterize the at least one abnormality in the scan, toassign one or more medical codes to the scan, to generate naturallanguage text data and/or a formatted report for the scan, and/or toautomatically generate other diagnosis data such as some or all ofdiagnosis data 440 based on the medical scan. Alternatively or inaddition, some inference functions can also be utilized to automaticallygenerate confidence score data 460, display parameter data 470, and/orsimilar scan data 480. The medical scan diagnosing system 108 cancompare the output preference to the output type data 612 of the medicalscan inference function 1105 to determine the selected inferencefunction 1105. For example, this can be used to decide between a firstmedical scan inference function that automatically generates medicalcodes and a second medical scan inference function that automaticallygenerates natural language text for medical reports based on the desiredtype of inference data 1110.

Prior to performing the selected medical scan inference function 1105,the medical scan diagnosing system 108 can automatically perform aninput quality assurance function 1106 to ensure the scan classifier data420 or other metadata of the medical scan accurately classifies themedical scan such that the appropriate medical scan inference function1105 of the appropriate scan category 1120 is selected. The inputquality assurance function can be trained on, for example, medical scanimage data 410 of plurality of previous medical scans with verified scancategories. Thus, the input quality assurance function 1106 can takemedical scan image data 410 as input and can generate an inferred scancategory as output. The inferred scan category can be compared to thescan category 1120 of the scan, and the input quality assurance function1106 can determine whether or not the scan category 1120 is appropriateby determining whether the scan category 1120 compares favorably to theautomatically generated inferred scan category. The input qualityassurance function 1106 can also be utilized to reassign the generatedinferred scan category to the scan category 1120 when the scan category1120 compares favorably to the automatically generated inferred scancategory. The input quality assurance function 1106 can also be utilizedto assign the generated inferred scan category to the scan category 1120for incoming medical scans that do not include any classifying data,and/or to add classifiers in scan classifier data 420 to medical scansmissing one or more classifiers.

Alternatively or in addition, performing the input quality assurancefunction 1106 can include performing a selected one of a plurality ofclassifier verification functions corresponding to the scan category1120 to determine a binary verification value indicating whether theassigned scan category 1120 is accurate. Each verification function canbe trained on a set of medical scans known to be correctly classifiedwith the corresponding scan category 1120, for example, where a kneex-ray verification function is trained on a plurality of scansclassified as “knee x-rays”, and in some embodiments, is also trained ona plurality of medical scans that are classified as “not knee x-rays.”In some embodiments, each verification function can be trained on themedical scan image data 410 and the binary indicator of whether or notthe scan is assigned to the corresponding scan category 1120.

In various embodiments, upon utilizing the input quality assurancefunction 1106 to determining that the scan category 1120 determined by ascan's scan classifier data 420 or other metadata is inaccurate, themedical scan diagnosing system 108 can transmit an alert and/or anautomatically generated inferred scan category to the medical entityindicating that the scan is incorrectly classified in the scanclassifier data 420 or other metadata. In some embodiments, the medicalscan diagnosing system 108 can automatically update performance scoredata 530 corresponding to the originating entity of the scan indicatedin originating entity data 423, or another user or entity responsiblefor classifying the scan, for example, where a lower performance scoreis generated in response to determining that the scan was incorrectlyclassified and/or where a higher performance score is generated inresponse to determining that the scan was correctly classified.

In some embodiments, the medical scan diagnosing system 108 can transmitthe medical scan and/or the automatically generated inferred scancategory to a selected user. The user can be presented the medical scanimage data 410 and/or other data of the medical scan via the interactiveinterface 275, for example, displayed in conjunction with the medicalscan assisted review system 102. The interface can prompt the user toindicate the appropriate scan category 1120 and/or prompt the user toconfirm and/or edit the inferred scan category, also presented to theuser. For example, scan review data 810 can be automatically generatedto reflect the user generated and/or verified scan category 1120, Thisuser indicated scan category 1120 can be utilized to select to themedical scan inference function 1105 and/or to update the scanclassifier data 420 or other metadata accordingly. In some embodiments,for example, where the scan review data 810 indicates that the selecteduser disagrees with the automatically generated inferred scan categorycreated by the input quality assurance function 1106, the medical scandiagnosing system 108 can automatically update performance score data630 of the input quality assurance function 1106 by generating a lowperformance score and/or determine to enter the remediation step 1140for the input quality assurance function 1106.

The medical scan diagnosing system 108 can also automatically perform anoutput quality assurance step 1107 after a medical scan inferencefunction 1105 has been performed on a medical scan to produce theinference data 1110, as illustrated in the embodiment presented in FIG.11B. The output quality assurance step 1107 can be utilized to ensurethat the selected medical scan inference function 1105 generatedappropriate inference data 1110 based on expert feedback. The inferencedata 1110 generated by performing the selected medical scan inferencefunction 1105 can be sent to a client device 120 of a selected expertuser, such as an expert user in the user database selected based oncategorized performance data 534 and/or qualification data 540 thatcorresponds to the scan category 1120 and/or the inference itself, forexample, by selecting an expert user best suited to review an identifiedabnormality classifier category 444 and/or abnormality pattern category446 in the inference data 1110 based on the categorized performance data534 and/or qualification data 540. The selected user can also correspondto a medical professional or other user employed at the originatingentity and/or corresponding to the originating medical professional,indicated in the originating entity data 423.

The medical scan and/or the inference data 1110 can be displayed to theselected expert user via the interactive interface 275 displayed by adisplay device corresponding to the client device 120 of the expertuser, for example, in conjunction with the medical scan assisted reviewsystem 102. Inference data 1110 can displayed in conjunction withmedical scan image data 410 as displayed annotation data 820 byutilizing one or more interface features indicated by interfacepreference data 560 of the expert user, the display parameter data 470of the medical scan, or other display parameters automatically generatedby the medical scan inference function 1105. In other embodiments, ablind review is utilized, and the inference data 1110 can be hidden fromthe expert user.

The client device can generate expert feedback data 1135, such as scanreview data 810, for transmission back to the medical scan diagnosingsystem 108, where the expert feedback data 1135 is generated based onuser input to the interactive interface entered in response to a promptdisplayed by the display device via the interactive interface 275 toprovide the expert feedback data by utilizing one or more interfacefeatures indicated by interface preference data 560 of the expert user,the display parameter data 470 of the medical scan, or other displayparameters automatically generated by the medical scan inferencefunction 1105. The expert feedback data 1135 can include indications ofwhether or not the inference data is correct and/or edits of theinference data 1110. In the case where a blind review is utilized,expert user can enter their own annotations without being biased by theinference data 1110 generated by the medical scan inference function1105, and the expert feedback data 1135 can include these expertannotations.

The medical scan diagnosing system 108 can evaluate the expert feedbackdata 1135 to generate inference accuracy data 1136, indicating whetheror not the inference data 1110 is accurate, by performing an inferencedata evaluation function 1138. If the inference data 1110 is determinedto be accurate, the diagnosis data 440 or other data corresponding tothe medical scan can be updated accordingly based on the inference data1110, and/or the inference data can be transmitted to the originatingentity or other requesting entity. If the inference data 1110 isdetermined to be inaccurate, the inference data can be automaticallymodified based on the expert feedback data received from the clientdevice, and the modified inference data 1110 and/or diagnosis dataindicated in the expert feedback data 1135 can be mapped to the medicalscan in the diagnosis data 440.

Inference accuracy data 1136 can also be based on a determineddiscrepancy determined by performing the inference data evaluationfunction 1138. For example, the inference accuracy data 1136 can bebased on a magnitude of the difference between the inference data 1110and the expert feedback data, and can be mapped to the medical scan. Theinference accuracy data 1136 can be further based on confidence data,determined based on an inference confidence score corresponding to theinference data 1110, for example, automatically generated by the medicalscan inference function 1105 in conjunction with generating theinference data 1110 and/or based on the performance score data 630 ofthe medical scan inference function 1105 that was utilized. Theinference accuracy data 1136 can be further based on a confidence scoregenerated for the expert review data and/or can be based on performancescore data 530 corresponding to the expert user.

In some embodiments, performing the inference data evaluation function1138 can include performing the annotation consensus function 1040 orutilizing other features of the medical scan annotator system 106 todetermine whether or not consensus is reached between the inference data1110 and the expert feedback data 1135, for example, by determiningwhether or not a calculated consensus confidence score 1050 exceeds aconsensus confidence threshold as described herein. The inferenceaccuracy data 1136 can be based on the consensus confidence score 1050.In some embodiments, inference data 1110 can be replaced with the finalconsensus annotation 1030 generated by performing the annotationconsensus function 1040. The final consensus annotation can utilizeweights, where the expert feedback data is weighted based on aconfidence score of the expert feedback data 1135 and/or the performancescore data 530 of the expert user, and/or where the inference data 1110is weighted based on the inference confidence score and/or theperformance score data 630 of the medical scan inference function 1105.

Alternatively or in addition, performing the inference data evaluationfunction 1138 can include comparing one or more binary values of expertannotations in the expert feedback data to corresponding binary valuesof the inference data 1110, such as “normal” and “abnormality detected”,and conclude that the medical scan diagnosis is accurate if the expert'sbinary values match the corresponding binary values of the inferencedata 1110. The inference data evaluation function 1138 can includecomparing indicated locations of detected abnormalities of the expertfeedback data and an abnormality location indicated in the automaticallygenerated inference data 1110, and determine the inference data isaccurate when the distance between these locations is within a thresholddistance. This calculated distance can be used to determine thediscrepancy level used to generate the inference accuracy data 1136. Theinference data evaluation function 1138 can include comparing an expertclassification of the detected abnormality with an automaticallygenerated classification of the detected abnormality in the inferencedata 1110 and determine that the inference data 1110 is accurate whenthe classification data matches. The inference data evaluation function1138 can include generating a first feature vector that includesfeatures of the scan review data 810 corresponding to the expertfeedback data, generating a second feature vector that includes featuresof the inference data 1110, and computing a Euclidian distance betweenthe two feature vectors, where the inference data 1110 is determined tobe accurate if a magnitude of the calculated Euclidian distance fallswithin a threshold Euclidian distance. This calculated Euclidiandistance can be used to determine the discrepancy level used to generatethe inference accuracy data 1136. In embodiments where the expertfeedback data 1135 is based on review of displayed inference data, theinference data evaluation function 1138 can include determining theinference data is accurate when the expert feedback data indicates thatthe diagnosis is correct.

In some embodiments, the inference data 1110 generated for all medicalscans processed by the medical scan diagnosing system 108 will undergothis output quality assurance step 1107 and be sent to a selectedexpert. Alternatively, a random or pseudo-random sampling of diagnosedmedical scans undergo the expert review, for example, based on a fixedoutput assurance percentage. Different originating entities can beassigned different output assurance percentages, for example, based on auser profile information such as subscription data 550 for the user orother usage limits assigned to the entity. Different inference data 1110can be assigned different output assurance percentages, for example,where every medical scan flagged as “not normal” is reviewed by anexpert. In such cases, a fixed percentage of medical scans flagged as“normal” can also be reviewed by a selected expert for additionalquality assurance. Different diagnosis can also be randomly orpseudo-randomly sampled based on different output assurance percentages,for example, where a higher proportion of medical scans with rarerand/or more severe diagnoses are reviewed than medical scans with lessrare and/or less severe diagnoses, for example, based on verificationpercentage of a medical code 447 of the inference data 1110, based onseverity and/or rarity of some or all of the inference data 1110, and/orbased on an ambiguity score and/or rarity score associated with an aliasmapping pair 925 utilized to generate the inference data 1110. In someembodiments, the inference confidence score can be utilized to determinewhether or not the inference data 1110 undergoes the output qualityassurance step 1107. For example, only inference data 1110 with aninference confidence score that fails to exceed a confidence scorethreshold is sent to the expert for review. The confidence scorethreshold can be the same or different for different originatingentities, the same or different for different scan categories, and/orthe same or different for different diagnosis types.

Performance score data 630 can be generated and/or updated for themedical scan inference function 1105 based on the inference accuracydata 1136 or other indication of whether or not the inference data 1110was accurate, where a high score is awarded for accurate inference dataand a low score is awarded for inaccurate inference data. Theperformance score data 630 can be further based on the determineddiscrepancy level between the inference data and the expert feedbackdata. The performance score data 630 can further be weighted based onthe severity of the diagnosis, where the performance score data ishigher for detected abnormalities that are more severe and/or rare thanless severe and/or rare. For example, performance score data is scoredlower for failing to detect abnormalities that are more severe than lesssevere. The performance score data 630 can further be based on whetherthe inaccuracy was a false positive diagnosis or a false negativediagnosis. For example, performance score data is scored lower for afalse negative diagnosis than a false positive diagnosis. Performancescore data 630 can also include determining efficiency of the medicalscan inference function 1105, for example, by computing a duration oftime to compute the inference data, amount of memory utilized, or otherefficiency data. Performance score data 530 can be generated and/orupdated for the expert user, for example, where a higher performancescore is award for higher discrepancy levels than lower discrepancylevels and/or where a higher performance score is awarded for detectingabnormalities that are more severe than less severe.

FIG. 11C illustrates an embodiment of the medical scan diagnosing system108 in conjunction with performing a remediation step 1140. The medicalscan diagnosing system 108 can monitor the performance of the set ofmedical scan inference functions 1105, for example, based on evaluatingthe inference accuracy data 1136 outputted by the inference dataevaluation function 1138 and/or based monitoring on the performancescore data 630 in the medical scan analysis function database, and candetermine whether or not if the corresponding medical scan inferencefunction 1105 is performing properly. This can include, for example,determining if a remediation step 1140 is necessary for a medical scaninference function 1105, for example, by comparing the performance scoredata 630 and/or inference accuracy data 1136 to remediation criteriadata 652. Determining if a remediation step 1140 is necessary can alsobe based on receiving an indication from the expert user or another userthat remediation is necessary for one or more identified medical scaninference functions 1105 and/or for all of the medical scan inferencefunctions 1105.

In various embodiments, a remediation evaluation function is utilized todetermine if a remediation step 1140 is necessary for medical scaninference function 1105. The remediation evaluation function can includedetermining that remediation is necessary when recent accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below the normal performance level of the particular inferencefunction. The remediation evaluation function can include determiningthat remediation is necessary when recent or overall accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below a recent or overall average for all or similar medicalscan inference functions 1105. The remediation evaluation function caninclude determining that remediation is necessary only after a thresholdnumber of incorrect diagnoses are made. In various embodiments, multiplethreshold number of incorrect diagnoses correspond to differentdiagnoses categories. For example, the threshold number of incorrectdiagnoses for remediation can be higher for false negative diagnosesthan false positive diagnoses. Similarly, categories corresponding todifferent diagnosis severities and/or rarities can have differentthresholds, for example where a threshold number of more severe and/ormore rare diagnoses that were inaccurate to necessitate remediation islower than a threshold number of less severe and/or less rare diagnosesthat were inaccurate.

The remediation step 1140 can include automatically updating anidentified medical inference function 1105. This can includeautomatically retraining identified medical inference function 1105 onthe same training set or on a new training set that includes new data,data with higher corresponding confidence scores, or data selected basedon new training set criteria. The identified medical inference function1105 can also be updated and/or changed based on the review datareceived from the client device. For example, the medical scan andexpert feedback data 1135 can be added to the training set of themedical inference function 1105, and the medical scan inference function1105 can be retrained on the updated training set. Alternatively or inaddition, the expert user can identify additional parameters and/orrules in the expert feedback data based on the errors made by theinference function in generating the inference data 1110 for the medicalscan, and these parameters and/or rules can be applied to update themedical scan inference function, for example, by updating the model typedata 622 and/or model parameter data 623.

The remediation step 1140 can also include determining to split a scancategory 1120 into two or more subcategories. Thus, two or more newmedical scan inference functions 1105 can be created, where each newmedical scan inference functions 1105 is trained on a correspondingtraining set that is a subset of the original training set and/orincludes new medical scan data corresponding to the subcategory. Thiscan allow medical scan inference functions 1105 to become morespecialized and/or allow functions to utilize characteristics and/ordiscrepancies specific to the subcategory when generating inference data1110. Similarly, a new scan category 1120 that was not previouslyrepresented by any of the medical scan inference functions 1105 can beadded in the remediation step, and a new medical scan inferencefunctions 1105 can be trained on a new set of medical scan data thatcorresponds to the new scan category 1120. Splitting a scan categoryand/or adding a scan category can be determined automatically by themedical scan diagnosing system 108 when performing the remediation step1140, for example, based on performance score data 630. This can also bedetermined based on receiving instructions to split a category and/oradd a new scan category from the expert user or other user of thesystem.

After a medical scan inference function 1105 is updated or created forthe first time, the remediation step 1140 can further undergo acommissioning test, which can include rigorous testing of the medicalscan inference function 1105 on a testing set, for example, based on thetraining parameters 620. For example, the commissioning test can bepassed when the medical scan inference function 1105 generates athreshold number of correct inference data 1110 and/or the test can bepassed if an overall or average discrepancy level between the inferencedata and the test data is below a set error threshold. The commissioningtest can also evaluate efficiency, where the medical scan inferencefunction 1105 only passes the commissioning test if it performs at orexceeds a threshold efficiency level. If the medical scan inferencefunction 1105 fails the commissioning test, the model type and/or modelparameters can be modified automatically or based on user input, and themedical scan inference function can be retested, continuing this processuntil the medical scan inference function 1105 passes the commissioningtest.

The remediation step 1140 can include decommissioning the medical scaninference function 1105, for example, while the medical scan inferencefunction is being retrained and/or is undergoing the commissioning test.Incoming scans to the medical scan diagnosing system 108 with a scancategory 1120 corresponding to a decommissioned medical scan interfacefunction 1105 can be sent directly to review by one or more users, forexample, in conjunction with the medical scan annotator system 106.These user-reviewed medical scans and corresponding annotations can beincluded in an updated training set used to train the decommissionedmedical scan inference function 1105 as part of the remediation step1140. In some embodiments, previous versions of the plurality of medicalscan image analysis functions can be stored in memory of the medicalscan diagnosing system and/or can be determined based on the versiondata 640 of a medical scan inference function 1105. A previous versionof a medical scan inference function 1105, such as most recent versionor version with the highest performance score, can be utilized duringthe remediation step 1140 as an alternative to sending all medical scansto user review.

In various embodiments, the output assurance threshold or other aspectsof the output quality assurance step 1107 are stricter for a rebuildingperiod of time after a medical scan inference function passes thecommissioning test. For example, all of, or a higher percentage of,inference data 1110 generated by the new and/or updated medical scaninference function 1105 can be reviewed by an expert user, and/or can bereviewed by a higher ranked expert user. The output quality assurancestep 1107 can be returned to normal after a fixed amount of time wherethe remediation step is not needed for the recommissioned medical scaninference function 1105, when the performance score data 630 exceeds acertain threshold, when a threshold number or proportion of accurateinference data 1110 is generated, or when other criteria is metindicating that the medical scan inference function is performingfavorably. In some embodiments, the medical scan inference functionautomatically is commissioned normally after passing the commissioningtest.

A medical scan image analysis function can also undergo the remediationstep 1140 automatically in response to a hardware and/or software updateon processing, memory, and/or other computing devices where the medicalscan inference function 1105 is stored and/or performed. Differentmedical scan inference functions 1105 can be containerized on their owndevices by utilizing a micro-service architecture, so hardware and/orsoftware updates may only necessitate that one of the medical scaninference functions 1105 undergo the remediation step 1140 while theothers remain unaffected. A medical scan inference function 1105 canalso undergo the remediation step 1140 automatically in response tonormal system boot-up, and/or periodically in fixed intervals. Forexample, in response to a scheduled or automatically detected hardwareand/or software update, change, or issue, one or more medical scan imageinference functions 1105 affected by this hardware or software can betaken out of commission until they each pass the commissioning test.Such criteria can be indicated in the remediation criteria data 652.

The medical scan diagnosing system 108 can automatically manage usagedata, subscription data, and/or billing data for the plurality of userscorresponding to user usage of the system, for example, by utilizing,generating, and/or updating some or all of the subscription data 550 ofthe user database. Users can pay for subscriptions to the system, whichcan include different subscription levels that can correspond todifferent costs. For example, a hospital can pay a monthly cost toautomatically diagnose up to 100 medical scans per month. The hospitalcan choose to upgrade their subscription or pay per-scan costs forautomatic diagnosing of additional scans received after the quota isreached and/or the medical scan diagnosing system 108 can automaticallysend medical scans received after the quota is reached to an expert userassociated with the hospital. In various embodiments incentive programscan be used by the medical scan diagnosing system to encourage expertsto review medical scans from different medical entities. For example, anexpert can receive credit to their account and/or subscription upgradesfor every medical scan reviewed, or after a threshold number of medicalscans are reviewed. The incentive programs can include interactions by auser with other subsystems, for example, based on contributions made tomedical scan entries via interaction with other subsystems.

FIG. 12A presents an embodiment of a medical scan interface featureevaluator system 110. A medical scan interface feature evaluator can beused evaluate proposed interface features or currently used interfacefeatures of an interactive interface 275 to present medical scans forreview by medical professionals or other users of one or more subsystems101.

In various embodiments, the medical scan interface feature evaluatorsystem 110 is operable to generate an ordered image-to-prompt mapping byselecting a set of user interface features to be displayed with each ofan ordered set of medical scans. The set of medical scans and theordered image-to-prompt mapping are transmitted to a set of clientdevices. A set of responses are generated by each client device inresponse to sequentially displaying each of the set of medical scans inconjunction with a mapped user interface feature indicated in theordered image-to-prompt mapping via a user interface. Response scoredata is generated by comparing each response to truth annotation data ofthe corresponding medical scan. Interface feature score datacorresponding to each user interface feature is generated based onaggregating the response score data, and is used to generate a rankingof the set of user interface features.

Users can receive medical scan data 1210 such as medical scan imageslices 410 and/or natural language text data 448 for one or moreselected medical scans with previously generated annotation or diagnosisdata, for example with high confidence score data 460 and/or a truthflag 461, or to be compared to expert review subsequently provided, forexample, received from an expert user in conjunction with the medicalscan annotation system 106 or another user identified as an expert inthe qualification data 540, for example, in conjunction with the medicalscan annotator system 106. Medical scan data 1210 for each scan can bepresented to selected users based on a selected subset of proposedinterface features 1205 which can include some or all interfacefeatures, for example, in conjunction with the medical scan assistedreview system 102, where known annotation data is hidden and/orpresented as displayed annotation data 820 via the selected subset ofproposed interface features 1205. The client device can generateresponse data 1220 based on user input via the interactive interface275. The response data 1220 can include scan review data 810, forexample, where each medical scan data 1210 is presented to the user inconjunction with the medical scan assisted review system 102. Themedical scan interface feature evaluator system 110 can perform aresponse evaluation function 1240 to generate aggregate interfacefeature performance data 1230 based on the response data 1220, forexample, where received annotations are compared to corresponding knownor later provided diagnosis data or other corresponding data of eachmedical scan. The aggregate interface feature performance data 1230 canbe used to score and/or rank the plurality of interface features, togenerating and/or evaluating performance score data 720, to generatedisplay parameter data 470 for medical scans, to automatically generateinterface preference data 560, and/or to determine default interfacefeatures for some or all of the subsystems 101. The medical scaninterface feature evaluating system 110 can utilize the medical scandatabase 342, user database 344, and/or interface feature database 348by communicating with the database storage system 140 via the network150, and/or can utilize another medical scan database, user database,and/or interface feature database stored in local memory.

Interactive interface features evaluated by the medical scan interfacefeature evaluator system can include textual and/or visual cues, menuoptions, response input methods, spatial interface layouts, some or allof the interface features, or other features of interactive interface275 as discussed herein in conjunction with one or more subsystems 101and/or store in interface feature database 348. Some or all of theinteractive interface features can include medical scan displayparameters as discussed herein, for example, indicated in displayparameter data 470 of each medical scan entry.

Textual and/or visual cues can include, for example, text-basedinterface features that instruct a user such as “click on a detectedabnormality” and/or ask questions of a user such as “what kind ofabnormality was detected?” Textual and visual cues can be used topresent annotations and/or diagnosis data that already correspond to thesystem for verification by the user, or to otherwise aid the user ingenerating their own annotation and/or diagnosis data. For example,annotations and/or diagnosis data previously provided by another medicalprofessional and/or generated automatically by the medical scaninterface feature evaluator system 110 or another system of the medicalscan processing system can be presented, where detected abnormalitiesare circled, highlighted, or otherwise visually indicated.

Menu options can include, for example, options to zoom in on selected orautomatically identified areas of interest, scroll through scan slices,automatically jump to the next slice that includes a detectedabnormality, view similar medical scans and their associated data, wherethe similar medical scans are automatically identified from a pluralityof medical scans and/or identified by a previous user, view past medicalscans associated with the same patient, automatically reveal and/or hidepreviously generated annotation and/or diagnosis data, or other optionsfor viewing the medical scan and associated data such as diagnosis data440 or other data of a medical scan entry 352.

Response input methods can include allowing users to provide annotationor diagnosis data by selecting one of a plurality of annotation options,click on an abnormality, outline a region that includes an abnormality,to select or upload similar medical scans or medical reports for similarcases, to shade in an abnormality, provide descriptive text and/or voicedata for the scan or one or more detected abnormalities, or otherresponse types.

Spatial interface layouts can include presenting one slide at a timewith a scroll bar, presenting a selected number of slices at once, forexample, as thumbnails, presenting a scan slice next to, or overlaidwith, annotation text or other diagnosis data, presenting the medicalscan side by side with a similar medical scan that is automaticallyselected and/or selected or uploaded by the user, presenting the medicalscan side by side with a past medical scan of the same patient, visuallyoverlaying or otherwise identifying changes in medical scans of the samepatient over time, etc. Herein, such textual and/or visual cues, menuoptions, response input methods, and/or spatial interface layouts willbe referred to interchangeably as “interface features”. Interfacefeatures can also include other interactive interface characteristicspresented and/or described herein in conjunction with other subsystemsof the medical scan processing system.

In various embodiments, a plurality of proposed interface features 1205are stored in the interface feature database 348 and/or a database inmemory of the medical scan interface feature evaluator system 110. Someor all of the plurality of proposed interface features can be receivedfrom one or more administrators of the medical scan interface featureevaluator system, and/or generated automatically by the medical scaninterface feature evaluator system based on previous aggregatestatistics of previously proposed interface features of the system. Allof the proposed interface features can undergo evaluation, or a subsetof proposed interface features can be selected, for example, manually byan administrator via the administrator interactive interface orautomatically by the medical scan interface feature evaluator systembased on received or automatically determined interface featurecriteria. A plurality of users of the system can also be selected. Forexample, users can be selected based on overall and/or categorizedperformance score data 530 of the user database 344 and/or otherwisegenerated and/or received by one or more subsystems of the medical scanprocessing system. Different users can be assigned a different set ofmedical scans from the subset randomly or pseudo-randomly and/or basedon the performance and/or specification data.

Each medical scan assigned to a user is further assigned one of theplurality of proposed interface features. This plurality ofscan-to-interface feature mappings 1225, designating a plurality of(scan, interface feature) pairs, can further be ordered by the medicalscan interface evaluating system based on an automatically generatedorder determined to be optimal for evaluating the interface features, orbased on ordering criteria designated by an administrator. The set ofproposed interface features 1205, medical scan data 1210, andscan-to-interface feature mappings 1225 are sent to client devicescorresponding to each user in a selected user set 1215. Data of eachmedical scan is presented to the client device of the corresponding usein accordance with the assigned proposed interface feature, for example,sequentially based on the determined order, via an interactive interfaceon a display device associated with the client device. The medical scaninterface feature evaluating system 110 receives responses correspondingto each displayed (scan, interface feature) pair in the response data1220 transmitted from each of the client devices entered by the user viathe interactive interface. In some embodiments, the medical scanassisted review system 102 is utilized in displaying the selectedinterface features on a user client device in conjunction with thecorresponding medical scans based on the scan-to-interface featuremapping 1225.

In various embodiments, a first user may receive the same or differentmedical scans than a second user. Some or all of the same medical scanssent to a first user and a second user can be paired to the same ordifferent proposed interface features. For example, a first user mayreceive scan X paired to interface feature A and scan Y paired tointerface feature B, and a second user may receive scan X paired tointerface feature B and scan Y paired to interface feature C. Some usersmay receive one or more of the same scans paired to multiple proposedinterface features. For example, a third user may receive scan Z pairedto interface feature I, then interface feature II, then interfacefeature III. The distribution of scans and interface features can bedetermined automatically to optimize the integrity of the response datareceived from the set of users to best aggregate scoring data for theinterface features and/or the users themselves.

The medical scan interface feature evaluating system 110 can performingthe response evaluation function 1240, for example, to score eachresponse by comparing the response to truth data, such as knowndiagnosis data or data entered by an expert, or other known data forexample, diagnosis data 440 with confidence score data 460 that includesa truth flag 461 or otherwise compares favorably to a truth threshold.Scoring each response can include determining if the diagnosis iscorrect or incorrect, for example, determining that a normal scan iscorrectly labeled as normal, or a scan that includes an abnormality isincorrectly labeled as normal, generating a first feature vector basedon the response data and a second feature vector based on the truth dataand calculating a Euclidian distance, and/or utilizing other performanceand/or accuracy measures described herein. Each response can also bescored for efficiency, for example, based on the amount of time takenfor a user to enter their annotations, diagnosis, or other relevantinput. In various embodiments, performing the response evaluationfunction can include performing part of all of the inference dataevaluation function 1138 and/or the annotation consensus function 1040.

In some embodiments, the response data can include interface featurefeedback data in response to data generated by the client device inresponse to an interface feature for the user to provide feedback, forexample, “On a scale from 1-5, how effective did you find this proposedinterface feature?” The score data can also be generated based on theparticular user, for example, based on user performance score data 530.For example, a response for an interface feature used with a chest x-rayreceived from a particular user that usually performs poorly ondiagnosing chest x-rays may be scored more highly if the responseincludes a correct diagnosis. Performing the response evaluationfunction 1240 can include scoring each response based on such userfeedback.

The medical scan interface evaluating system can generate aggregateinterface feature performance data 1230 based on individual scores foreach response. The aggregate interface feature performance data 1230 caninclude scoring and/or ranking data for each of the proposed interfacefeatures, which can be used to generate and/or update performance scoredata 720. The aggregate interface feature performance data 1230 canfurther be used to generate and/or update performance score data 530 foreach selected user from whom responses data was received and evaluated.For example, all of the performance scores across multiple users thatviewed interface feature I can be averaged or otherwise aggregated, andcan be compared to an aggregate score generated based on all of thescores across multiple users that viewed interface feature II.

The aggregate interface feature performance data 1230 can includecategorized aggregate data, where aggregate data for each interfacefeature can be further broken down based on scores for distinct scancategories, for example, based on the scan classifier data 420, forexample, where a first aggregate data score is generated for interfacefeature I based on scores from all knee x-rays, and a second aggregatedata score is generated for interface feature I based on scores from allchest x-rays. Aggregate data for each interface feature can be furtherbased on scores for distinct diagnosis categories, where a firstaggregate data score is generated for interface feature I based onscores from all normal scans, and a second aggregate data score isgenerated for interface feature I based on scores from all scans thatcontain an abnormality. This can be further broken down, where a firstaggregate score is generated for interface feature I based on all scoresfrom scans that contain an abnormality of a first type and/or in a firstanatomical location, and a second aggregate score is generated forinterface feature I based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each interface feature can be further based on scores for distinctuser types, where a first aggregate data score is generated forinterface feature I based on scores from all novice users and/or userswith low performance scores, and a second aggregate data score isgenerated for interface feature I based on scores from all expert usersand/or users with high performance scores. Aggregate data for eachinterface feature can be further based on scores for distinct medicalentities, where a first aggregate data score is generated for interfacefeature I based on scores from a first hospital, and a second aggregatedata score is generated for interface feature I based on scores from asecond hospital.

The aggregate interface feature performance data 1230 can be sent to anadministrator client device for display by a corresponding administratorand/or can be stored in the database of interface features database 348,where interface feature scores are mapped to their correspondinginterface feature in the database in the performance score data 720. Theaggregate interface feature performance data 1230 can be used to rankthe interface features holistically, and/or rank the interface featuresamong different scan type and/or user/entity type categories. Themedical scan interface feature evaluator system 110 can providerecommendations to an administrator for which proposed interfacefeatures should be used in practice to aid users in annotating,diagnosing, and/or otherwise reviewing new and/or triaged medical scans.The same and/or different interface features can be recommended for useby different users, different hospitals, and/or different scan typesbased on the aggregate interface feature performance data 1230. In someembodiments, these recommendations are automatically processed by themedical scan processing system 100 to determine which proposed interfacefeatures are used by other subsystems 101, and/or the other subsystemscan automatically select a highly ranked interface feature in aparticular scan-based and/or user-based category for display by itsinteractive interface along with medical scan data. In some embodiments,interface preference data 560 can be generated automatically for some orall of the users based on the aggregate interface feature performancedata 1230 for storage in the user database. This can be used tocustomize interface features utilized by in some or all of thesubsystems 101 for particular users and/or medical entities, such asparticular medical professionals and/or particular hospitals.

The aggregate interface feature performance data 1230 can be also beused by the medical scan interface feature evaluator system 110 toautomatically remove and/or modify some or all of the proposed interfacefeatures 1205, and/or to automatically generate new proposed interfacefeatures 1205. The medical scan interface feature evaluator data canmake such removal, modification, and/or new proposed interface featurerecommendations to an administrator. The aggregate interface featureperformance data 1230 can be also be used by the medical scan interfacefeature evaluator system 110 to automatically select a new set ofproposed interface features and/or scan-to-interface feature mappings tobe sent to the same or different users, where the new medical scansand/or user set can also be selected based on aggregate interfacefeature performance data 1230. This process can be repeated multipletimes to narrow the proposed interface features 1205, tweak, adjust orotherwise modify the proposed interface features 1205, finalize theproposed interface features 1205, gain further insights in specificcategories, evaluate interface features for a new type of users or newtype of scan, periodically check that interface features being used byother subsystems remain current and optimal, or to otherwise determinenew interface feature rankings.

The aggregate interface feature performance data 1230 can be used togenerate some or all of the user performance score data 530 and/or tootherwise rank users and/or hospitals. While many of the proposedinterface features may be in testing mode, some interface features maybe ranked highly enough to score user and/or hospital performance. Forexample, if 95% of users score highly on medical scans presented withinterface feature A, and a first user consistently receives low scoreson medical scans presented with interface feature A, the medical scaninterface evaluating system can use this data to determine that thefirst user has a low performance score and/or low rank. This can be usedby the medical scan interface evaluating system to automatically removelow ranked users from future sets of users selected when evaluating newproposed interface features. This information can also be stored in auser database used by other subsystems or otherwise can be used by othersubsystems in conjunction with other user performance tracking asdescribed herein.

FIGS. 12B-12C present example embodiments of an interactive interfacepresented in conjunction with the medical scan interface featureevaluator system 110. In an example embodiment, three types of interfacefeatures are tested using wrist x-rays. A first interface featurepresents a wrist x-ray and includes no annotation data. A secondinterface feature, as shown in FIG. 12B, presents hand-generated regionsof interest 1290 encircling suspected sites of fracture in conjunctionwith presenting the wrist x-ray. The third interface feature, as shownin FIG. 12C, includes a marquee listing bones suspicious for fractureand a measure of confidence in that judgement in conjunction withpresenting the wrist x-ray. In the example displayed in FIG. 12C,metacarpal and scaphoid are listed as suspicious for fracture, with theconfidence presented as one out of four blocks shaded.

FIG. 13 presents an embodiment of a medical scan image analysis system112. A medical scan image analysis system 112 can be used to generateand/or perform one or more medical scan image analysis functions byutilizing a computer vision-based learning algorithm on a training setof medical scans with known annotation data, diagnosis data, labelingand/or medical code data, report data, patient history data, patientrisk factor data, and/or other metadata associated with medical scans.These medical scan image analysis functions can be used to generateinference data for new medical scans that are triaged or otherwiserequire inferred annotation data, diagnosis data, labeling and/ormedical code data, and/or report data. For example, some medical scanimage analysis functions can correspond to medical scan inferencefunctions of the medical scan diagnosing system or other medical scananalysis functions of the medical scan analysis function database 348.The medical scan image analysis functions can be used to determinewhether or not a medical scan is normal, to detect the location of anabnormality in one or more slices of a medical scan, and/or tocharacterize a detected abnormality. The medical scan image analysissystem can be used to generate and/or perform computer vision basedmedical scan image analysis functions utilized by other subsystems ofthe medical scan processing system as described herein, aiding medicalprofessionals to diagnose patients and/or to generate further data andmodels to characterize medical scans. The medical scan image analysissystem can include a processing system that includes a processor and amemory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations.

In various embodiments, the medical scan image analysis system 112 isoperable to receive a plurality of medical scans that represent athree-dimensional anatomical region and include a plurality ofcross-sectional image slices. A plurality of three-dimensionalsubregions corresponding to each of the plurality of medical scans aregenerated by selecting a proper subset of the plurality ofcross-sectional image slices from each medical scan, and by furtherselecting a two-dimensional subregion from each proper subset ofcross-sectional image slices. A learning algorithm is performed on theplurality of three-dimensional subregions to generate a fullyconvolutional neural network. Inference data corresponding to a newmedical scan received via the network is generated by performing aninference algorithm on the new medical scan by utilizing the fullyconvolutional neural network. An inferred abnormality is identified inthe new medical scan based on the inference data.

A training set of medical scans used to train one more medical scanimage analysis functions can be received from one or more client devicesvia the network and/or can be retrieved from the medical scan database342, for example, based on training set data 621 corresponding tomedical scan image analysis functions. Training set criteria, forexample, identified in training parameters 620 of the medical scan imageanalysis function, can be utilized to automatically identify and selectmedical scans to be included in the training set from a plurality ofavailable medical scans. The training set criteria can be automaticallygenerated based on, for example, previously learned criteria, and/ortraining set criteria can be received via the network, for example, froman administrator of the medical scan image analysis system. The trainingset criteria can include a minimum training set size. The training setcriteria can include data integrity requirements for medical scans inthe training set such as requiring that the medical scan is assigned atruth flag 461, requiring that performance score data 530 for a hospitaland/or medical professional associated with the medical scan comparesfavorably to a performance score threshold, requiring that the medicalscan has been reviewed by at least a threshold number of medicalprofessionals, requiring that the medical scan and/or a diagnosiscorresponding to a patient file of the medical scan is older than athreshold elapsed time period, or based on other criteria intended toinsure that the medical scans and associated data in the training set isreliable enough to be considered “truth” data. The training set criteriacan include longitudinal requirements such the number of requiredsubsequent medical scans for the patient, multiple required types ofadditional scans for the patient, and/or other patient filerequirements.

The training set criteria can include quota and/or proportionrequirements for one or more medical scan classification data. Forexample, the training set criteria can include meeting quota and/orproportion requirements for one or more scan types and/or human bodylocation of scans, meeting quota or proportion requirements for a numberof normal medical scans and a number of medical scans with identifiedabnormalities, meeting quota and/or proportion requirements for a numberof medical scans with abnormalities in certain locations and/or a numberof medical scans with abnormalities that meet certain size, type, orother characteristics, meeting quota and/or proportion data for a numberof medical scans with certain diagnosis or certain corresponding medicalcodes, and/or meeting other identified quota and/or proportion datarelating to metadata, patient data, or other data associated with themedical scans.

In some embodiments, multiple training sets are created to generatecorresponding medical scan image analysis functions, for example,corresponding to some or all of the set of medical scan inferencefunctions 1105. Some or all training sets can be categorized based onsome or all of the scan classifier data 420 as described in conjunctionwith the medical scan diagnosing system 108, where medical scans areincluded in a training set based on their scan classifier data 420matching the scan category of the training set. In some embodiments, theinput quality assurance function 1106 or another input check step can beperformed on medical scans selected for each training set to confirmthat their corresponding scan classifier data 420 is correct. In someembodiments, the input quality assurance function can correspond to itsown medical scan image analysis function, trained by the medical scanimage analysis system, where the input quality assurance functionutilizes high level computer vision technology to determine a scancategory 1120 and/or to confirm the scan classifier data 420 alreadyassigned to the medical scan.

In some embodiments, the training set will be used to create a singleneural network model, or other model corresponding to model type data622 and/or model parameter data 623 of the medical scan image analysisfunction that can be trained on some or all of the medical scanclassification data described above and/or other metadata, patient data,or other data associated with the medical scans. In other embodiments, aplurality of training sets will be created to generate a plurality ofcorresponding neural network models, where the multiple training setsare divided based on some or all of the medical scan classification datadescribed above and/or other metadata, patient data, or other dataassociated with the medical scans. Each of the plurality of neuralnetwork models can be generated based on the same or different learningalgorithm that utilizes the same or different features of the medicalscans in the corresponding one of the plurality of training sets. Themedical scan classifications selected to segregate the medical scansinto multiple training sets can be received via the network, for examplebased on input to an administrator client device from an administrator.The medical scan classifications selected to segregate the medical scanscan be automatically determined by the medical scan image analysissystem, for example, where an unsupervised clustering algorithm isapplied to the original training set to determine appropriate medicalscan classifications based on the output of the unsupervised clusteringalgorithm.

In embodiments where the medical scan image analysis system is used inconjunction with the medical scan diagnosing system, each of the medicalscan image analysis functions associated with each neural network modelcan correspond to one of the plurality of neural network modelsgenerated by the medical scan image analysis system. For example, eachof the plurality of neural network models can be trained on a trainingset classified on scan type, scan human body location, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data as discussed in conjunction with the medical scandiagnosing system. In embodiments where the training set classifiers arelearned, the medical scan diagnosing system can determine which of themedical scan image analysis functions should be applied based on thelearned classifying criteria used to segregate the original trainingset.

A computer vision-based learning algorithm used to create each neuralnetwork model can include selecting a three-dimensional subregion 1310for each medical scan in the training set. This three-dimensionalsubregion 1310 can correspond to a region that is “sampled” from theentire scan that may represent a small fraction of the entire scan.Recall that a medical scan can include a plurality of orderedcross-sectional image slices. Selecting a three-dimensional subregion1310 can be accomplished by selecting a proper image slice subset 1320of the plurality of cross-sectional image slices from each of theplurality of medical scans, and by further selecting a two-dimensionalsubregion 1330 from each of the selected subset of cross-sectional imageslices of the each of the medical scans. In some embodiments, theselected image slices can include one or more non-consecutive imageslices and thus a plurality of disconnected three-dimensional subregionswill be created. In other embodiments, the selected proper subset of theplurality of image slices correspond to a set of consecutive imageslices, as to ensure that a single, connected three-dimensionalsubregion is selected.

Consider the case where the selected image slices correspond to a set ofconsecutive image slices. Selecting the two-dimensional subregions 1330from each image slice in the of a scan can include selecting the sameregion of pixels or a region determined from same vertical andhorizontal borders to generate a three-dimensional subregion thatcorresponds to a prism shape. For example, selecting the same-sizedsquare region centered at the same (x,y) coordinate pair of each of theimage slices of a scan would result in a three-dimensional subregionthat corresponds to a rectangular prism, and selecting the same-sizedcircular region centered at the same (x,y) coordinate pair of each ofthe image slices would result in a three-dimensional subregion thatcorresponds to a cylindrical shape. In other embodiments, differentsized and/or shaped two-dimensional subregions can be selected from eachimage slice of the scan. For example, different sized circular regionscan be selected that are centered at the same (x,y) coordinate pair ofeach of the image slices, where the largest circle is selected from thecenter image slice in the set of ordered consecutive slices, where thesmallest circles are selected from the first and last image slices inthe set of ordered consecutive slices, where the circles selected fromthe first slice to the center slice monotonically increase in size, andwhere the circles selected from the center slice to the last slicemonotonically decrease in size. Such as strategy would result in athree-dimensional subregion that resembles a sphere.

Each three-dimensional region selected from each of the plurality ofmedical scans in the training set can correspond to the same size ordifferent sizes, and can correspond to the same shape or differentshapes. If different shapes and/or sizes are selected for differentmedical scans, the shape and/or size can be determined randomly orpseudo-randomly and/or deterministically based on characteristics of thescan itself, such as based on the size and/or shape of a knownabnormality 1340 in the scan. For example, in some embodiments, thethree-dimensional region can be selected to include a known abnormality.The known abnormality 1340 can be closely “surrounded” by a subregion,for example, where the diameter of a spherical subregion is selectedbased on a maximum distance between any two boundary points of the knownabnormality. The known abnormality 1340 can also be completely“outlined” by the subregion, where the shape of each two-dimensionalregion is determined based on tracing a perimeter of the knownabnormality in each image slice.

In some embodiments, an ideal shape and/or size for thethree-dimensional subregions can be identified. The identified idealshape and/or size can include a single shape and/or size, or a set ofideal shapes and/or a range of ideal sizes. The identified ideal shapeand/or size can include a range for the total number of pixels, a rangefor the total number of bytes, a range for the total number of imageslices, a range for the number of pixels in each two-dimensionalsubregion, a range for a diameter of circular subregions, a horizontaland/or vertical range for two-dimensional subregions, a set oftwo-dimensional shapes, or other shape and/or size criteria. The idealshape and/or size requirements can be based on system requirements suchas processing restrictions, memory restrictions, efficiencyrequirements, and/or computation time requirements, as processingsubregions that are too large may not be reasonable the software and/orhardware available to the medical scan image analysis system.Alternatively or in addition, the ideal shape and/or size requirementscan be based on model accuracy requirements, as building a neuralnetwork model based on regions that are too small may result in a poormodel that results in inaccurate inferences by the model on new medicalscans. The identified ideal shape and/or size can be based onrequirements received via the network, for example, from anadministrator client device based on user input by an administrator. Theidentified ideal shape and/or size can also be generated automaticallyby the medical scan image analysis system, for example, by calculating amaximum region size based on known or determined system requirementsand/or by generating multiple models that utilized different shapesand/or sizes and determining the ideal shape and/or size based ontesting the multiple models and determining which of the multiple modelsachieve a desired model accuracy.

Selecting the three-dimensional subregion also includes determining alocation for selection. In some embodiments, the location of thethree-dimensional subregion is based on a uniform distribution. In otherembodiments, the location of the three-dimensional subregion is selectedbased on a non-uniform distribution. For example, selecting thethree-dimensional subregion can be based on the location of a knownabnormality 1340. As discussed previously, the three-dimensionalsubregion can be selected to include the entire known abnormality 1340.The three-dimensional subregion can also be selected to include at leasta part of the known abnormality. The location of the three-dimensionalsubregion can be deterministic, for example, where a center of theabnormality is centered in the three-dimensional subregion.

In various embodiments, a probability distribution function (PDF) can beused to determine the center location of the subregion, where selectinga subregion that is closer to the center to the subregion is moreprobable than selecting a subregion that is further from the center ofthe subregion. FIG. 13B presents an example embodiment of utilizing anx-axis PDF 1342, centered at a central x coordinate of the abnormality1340, to select an x coordinate, and utilizing a y-axis PDF 1344,centered at a central y coordinate of the abnormality 1340, to select ay coordinate, where the selected x and y coordinates are used todetermine the location of the subregion. In this fashion, consider afirst possible subregion centered at a first location that is a firstdistance away from a center or nearest border of the known abnormality,and a second possible subregion centered at a second location that is asecond distance away from the center or nearest border of the knownabnormality. The PDF could dictate that selecting the first possiblesubregion is more probable than selecting the second possible subregionas a result of the first distance being shorter than the seconddistance. The two-dimensional subregions can be selected based on theideal two-dimensional subregion size and/or shape requirements and theselected center (x,y) coordinate pair. In embodiments where the idealtwo-dimensional subregion size requirements includes a size range, thesubregion will further be determined based on probabilities assigned toeach of the sizes in the range. In embodiments where the idealtwo-dimensional subregion shape requirements includes a set of shapesand/or a set of shape parameters, the subregion will further bedetermined based on based on probabilities assigned to each of theshapes in the set of shapes and/or probabilities assigned to values ofeach parameter in the set of shape parameters.

Similarly, as illustrated in FIG. 13C, a center image slice of theselected proper image slice subset 1320 can be based on a slice PDF1346, for example, centered about a central slice that includes theknown abnormality in the plurality of ordered image slices. Theremainder of the image slices will be determined based on the idealslice number requirements and the center image slice. In embodimentswhere the ideal slice number requirement includes a range, the remainderof the image slices can be further determined based on probabilitiesassigned to each number of slices in the ideal slice number range orbased on other deterministic factors.

Each PDF, such as PDFs 1342, 1344, and 1346, can be based on a Gaussiandistribution or other distribution centered at the location of theabnormality with three random or pseudo-random variables: the center ofthe consecutive subset of image slices and the center (x,y) coordinatepair for the two-dimensional subregions of each image slice. In variousembodiments, the PDF model and/or model parameters are determined basedon received PDF data. For example, the received PDF parameter data caninclude the PDF itself, can indicate that a Gaussian distribution with acenter slice variance parameter of 3 slices, with an x varianceparameter of 10 pixels, and a y variance parameter of 10 pixels. The PDFmodel and/or parameters of a selected PDF model can also beautomatically generated by the medical scan image analysis system, forexample, after testing models generated utilizing different PDFs todetermine the model that produces the most accurate inferences.

In some embodiments, all possible subregion locations are assigned anon-zero probability. In other embodiments, some locations are assigneda probability of zero, and thus subregions corresponding to suchlocations will not be selected. For example, all subregions at locationsthat do not include the abnormality are assigned a probability of zeroand are never sampled. In some embodiments, subregions at locations thatdo not correspond to a region of interest, for example, locations thatdo not correspond to a location of the human body corresponding to theneural network model to be created can be assigned a probability ofzero. For example, if the training set is created to generate a neuralnetwork model for detecting abnormalities in the lung, regions of eachcross-sectional image slice that do not include the lung, or entireslices for example at the beginning or end of the scan that do notinclude the lung, can be assigned a probability of zero. This maskingstep can include determining regions of the scan that do not correspondto the region of interest, for example, by applying another computervision-based masking function. The computer vision-based maskingfunction can be generated automatically by the medical scan imageanalysis system or other system of the medical scan processing system,for example, where the computer vision-based masking function is trainedon the training data, for example, where regions to be masked areidentified manually in the training data.

In some embodiments, a density windowing step can be applied to the fullscan or the selected three-dimensional subregion. The density windowingstep can include utilizing a selected upper density value cut off and/ora selected lower density value cut off, and masking pixels with highervalues than the upper density value cut off and/or masking pixels withlower values than the lower density value cut off. The upper densityvalue cut off and/or a selected lower density value cut off can bedetermined based on based on the range of density values included in theregion that includes the abnormality, and/or based on the range ofdensity values associated with the abnormality itself, based on userinput to a subsystem, based on display parameter data associated withthe medical scan or associated with medical scans of the same type,and/or can be learned in the training step.

Having determined the subregion training set 1315 of three-dimensionalsubregions 1310 corresponding to the set of full medical scans in thetraining set, the medical scan image analysis system can complete atraining step 1352 by performing a learning algorithm on the pluralityof three-dimensional subregions to generate model parameter data 1355 ofa corresponding learning model. While a convolutional neural network isa preferred embodiment, the learning model can additionally oralternatively include one or more of a Bayesian model, a support vectormachine model, a cluster analysis model, or other supervised orunsupervised learning model. The model parameter data 1355 generated byperforming the learning algorithm 1350, and the model parameter data1355 can be utilized to determine the corresponding medical scan imageanalysis functions. For example, some or all of the model parameter data1355 can be mapped to the medical scan analysis function in the modelparameter data 623 or can otherwise define the medical scan analysisfunction.

The training step 1352 can include creating feature vectors for eachthree-dimensional subregion of the training set for use by the learningalgorithm 1350 to generate the model parameter data 1355. The featurevectors can include the pixel data of the three-dimensional subregionssuch as density values and/or grayscale values of each pixel based on adetermined density window. The feature vectors can also include otherfeatures as additional input features or desired output features, suchas known abnormality data such as location and/or classification data,patient history data such as risk factor data or previous medical scans,diagnosis data, responsible medical entity data, scan machinery model orcalibration data, contrast agent data, medical code data, annotationdata that can include raw or processed natural language text data, scantype and/or anatomical region data, or other data associated with theimage, such as some or all data of a medical scan entry 352. Featurescan be selected based on administrator instructions received via thenetwork and/or can be determined based on determining a feature set thatreduces error in classifying error, for example, by performing across-validation step on multiple models created using different featuresets. The feature vector can be split into an input feature vector andoutput feature vector. The input feature vector can include data thatwill be available in subsequent medical scan input, which can includefor example, the three-dimensional subregion pixel data and/or patienthistory data. The output feature vector can include data that will beinferred in subsequent medical scan input and can include single outputvalue, such as a binary value indicating whether or not the medical scanincludes an abnormality or a value corresponding to one of a pluralityof medical codes corresponding to the image. The output feature vectorcan also include multiple values which can include abnormality locationand/or classification data, diagnosis data, or other output. The outputfeature vector can also include a determined upper density value cut offand/or lower density value cut off, for example, characterizing whichpixel values were relevant to detecting and/or classifying anabnormality. Features included in the output feature vector can beselected to include features that are known in the training set, but maynot be known in subsequent medical scans such as triaged scans to bediagnosed by the medical scan diagnosing system, and/or scans to belabeled by the medical scan report labeling system. The set of featuresin the input feature vector and output feature vector, as well as theimportance of different features where each feature is assigned acorresponding weight, can also be designated in the model parameter data1355.

Consider a medical scan image analysis function that utilizes a neuralnetwork. The neural network can include a plurality of layers, whereeach layer includes a plurality of neural nodes. Each node in one layercan have a connection to some or all nodes in the next layer, where eachconnection is defined by a weight value. Thus, the model parameter data1355 can include a weight vector that includes weight values for everyconnection in the network. Alternatively or in addition, the modelparameter data 1355 can include any vector or set of parametersassociated with the neural network model, which can include an upperdensity value cut off and/or lower density value cut off used to masksome of the pixel data of an incoming image, kernel values, filterparameters, bias parameters, and/or parameters characterizing one ormore of a plurality of convolution functions of the neural networkmodel. The medical scan image analysis function can be utilized toproduce the output vector as a function of the input feature vector andthe model parameter data 1355 that characterizes the neural networkmodel. In particular, the medical scan image analysis function caninclude performing a forward propagation step plurality of neuralnetwork layers to produce an inferred output vector based on the weightvector or other model parameter data 1355. Thus, the learning step 1405utilized in conjunction with a neural network model can includedetermining the model parameter data 1355 corresponding to the neuralnetwork model, for example, by populating the weight vector with optimalweights that best reduce output error.

In particular, determining the model parameter data 1355 can includeutilizing a backpropagation strategy. The forward propagation algorithmcan be performed on at least one input feature vector corresponding toat least one medical scan in the training set to propagate the at leastone input feature vector through the plurality of neural network layersbased on initial and/or default model parameter data 1355, such as aninitial weight vector of initial weight values set by an administratoror chosen at random. The at least one output vector generated byperforming the forward propagation algorithm on the at least one inputfeature vector can be compared to the corresponding at least one knownoutput feature vector to determine an output error. Determining theoutput error can include, for example, computing a vector distance suchas the Euclidian distance, or squared Euclidian distance, between theproduced output vector and the known output vector, and/or determiningan average output error such as an average Euclidian distance or squaredEuclidian distance if multiple input feature vectors were employed.Next, gradient descent can be performed to determine an updated weightvector based on the output error or average output error. This gradientdescent step can include computing partial derivatives for the errorwith respect to each weight, or other parameter in the model parameterdata 1355, at each layer starting with the output layer. Chain rule canbe utilized to iteratively compute the gradient with respect to eachweight or parameter at each previous layer until all weight's gradientsare computed. Next updated weights, or other parameters in the modelparameter data 1355, are generated by updating each weight based on itscorresponding calculated gradient. This process can be repeated on atleast one input feature vector, which can include the same or differentat least one feature vector used in the previous iteration, based on theupdated weight vector and/or other updated parameters in the modelparameter data 1355 to create a new updated weight vector and/or othernew updated parameters in the model parameter data 1355. This processcan continue to repeat until the output error converges, the outputerror is within a certain error threshold, or another criterion isreached to determine the most recently updated weight vector and/orother model parameter data 1355 is optimal or otherwise determined forselection.

Having determined the medical scan neural network and its final othermodel parameter data 1355, an inference step 1354 can be performed onnew medical scans to produce inference data 1370, such as inferredoutput vectors. The inference step can include performing the forwardpropagation algorithm to propagate an input feature vector through aplurality of neural network layers based on the final model parameterdata 1355, such as the weight values of the final weight vector, toproduce the inference data. This inference step 1354 can correspond toperforming the medical scan image analysis function, as defined by thefinal model parameter data 1355, on new medical scans to generate theinference data 1370, for example, in conjunction with the medical scandiagnosing system 108 to generate inferred diagnosis data or otherselected output data for triaged medical scans based on itscorresponding the input feature vector.

The inference step 1354 can include applying the density windowing stepto new medical scans. If the training step 1352 was used to determineoptimal upper density value cut off and/or lower density value cut offvalues to designate an optimal density window, the inference step 1354can include masking pixels of incoming scans that fall outside of thisdetermined density window before applying the forward propagationalgorithm. Similarly, if learned parameters of one or more convolutionalfunctions correspond to the optimal upper density value cut off and/orlower density value cut off values, the density windowing step isinherently applied when the forward propagation algorithm is performedon the new medical scans.

In some embodiments where a medical scan analysis function is defined bymodel parameter data 1355 corresponding to a neutral network model, theneural network model can be a fully convolutional neural network. Insuch embodiments, only convolution functions are performed to propagatethe input feature vector through the layers of the neural network in theforward propagation algorithm. This enables the medical scan imageanalysis functions to process input feature vectors of any size. Forexample, as discussed herein, the pixel data corresponding to thethree-dimensional subregions is utilized input to the forwardpropagation algorithm when the training step 1352 is employed topopulate the weight vector and/or other model parameter data 1355.However, when performing the forward propagation algorithm in theinference step 1354, the pixel data of full medical scans can beutilized as input, allowing the entire scan to be processed to detectand/or classify abnormalities, or otherwise generate the inference data1370. This may be a preferred embodiment over other embodiments wherenew scans must also be sampled by selecting a three-dimensionalsubregions and/or other embodiments where the inference step requires“piecing together” inference data 1370 corresponding to multiplethree-dimensional subregions processed separately.

In embodiments that utilize a fully convolutional neural network, paddeddata can be generated for each of the plurality of three-dimensionalsubregions in the training step 1352 and/or for each full medical scanin the inference step 1354 to allow the convolution functions to beapplied on all portions of the subregion or full medical scan during theforward propagation algorithm. For example, padded data for eachtwo-dimensional region, for example, at each of for boundaries of arectangular region of the three-dimensional subregion, can be generated,and/or padded slices can be generated to be included before and/or afterthe set of consecutive slices. The amount of padded data to be generatedat each two-dimensional boundary, or the number of padded slices to begenerated before and/or after the consecutive slices can be based onparameters of the convolutional function utilized in the forwardpropagation step. The padded data can include zero-padded data. In someembodiments, the padded data can be generated based on a data reflectionat a plurality of boundaries of the three-dimensional subregion or thefull medical scan. Generating padded data based on data reflectionrather than zero-padding can create more natural, seamless boundariesthat are more contextually appropriate, and can better emulate actualimage data at these boundaries.

The inferred output vector of the inference data 1370 can include aplurality of abnormality probabilities mapped to a pixel location ofeach of a plurality of cross-sectional image slices of the new medicalscan. For example, the inferred output vector can indicate a set ofprobability matrices 1371, where each matrix in the set corresponds toone of the plurality of image slices of the medical scan, where eachmatrix is a size corresponding to the number of pixels in each imageslice, where each cell of each matrix corresponds to a pixel of thecorresponding image slice, whose value is the abnormality probability ofthe corresponding pixel.

A detection step 1372 can include determining if an abnormality ispresent in the medical scan based on the plurality of abnormalityprobabilities. Determining if an abnormality is present can include, forexample, determining that a cluster of pixels in the same region of themedical scan correspond to high abnormality probabilities, for example,where a threshold proportion of abnormality probabilities must meet orexceed a threshold abnormality probability, where an average abnormalityprobability of pixels in the region must meet or exceed a thresholdabnormality probability, where the region that includes the cluster ofpixels must be at least a certain size, etc. Determining if anabnormality is present can also include calculating a confidence scorebased on the abnormality probabilities and/or other data correspondingto the medical scan such as patient history data. The location of thedetected abnormality can be determined in the detection step 1372 basedon the location of the pixels with the high abnormality probabilities.The detection step can further include determining an abnormality region1373, such as a two-dimensional subregion on one or more image slicesthat includes some or all of the abnormality. The abnormality region1373 determined in the detection step 1372 can be mapped to the medicalscan to populate some or all of the abnormality location data 443 foruse by one or more other subsystems 101 and/or client devices 120.Furthermore, determining whether or not an abnormality exists in thedetection step 1372 can be used to populate some or all of the diagnosisdata 440 of the medical scan, for example, to indicate that the scan isnormal or contains an abnormality in the diagnosis data 440.

An abnormality classification step 1374 can be performed on a medicalscan in response to determining an abnormality is present.Classification data 1375 corresponding to one or more classificationcategories such as abnormality size, volume, pre-post contract, doublingtime, calcification, components, smoothness, texture, diagnosis data,one or more medical codes, a malignancy rating such as a Lung-RADSscore, or other classifying data as described herein can be determinedbased on the detected abnormality. The classification data 1375generated by the abnormality classification step 1374 can be mapped tothe medical scan to populate some or all of the abnormalityclassification data 445 of the corresponding abnormality classifiercategories 444 and/or abnormality pattern categories 446 and/or todetermine one or more medical codes 447 of the medical scan. Theabnormality classification step 1374 can include performing anabnormality classification function on the full medical scan, or theabnormality region 1373 determined in the detection step 1372. Theabnormality classification function can be based on another modeltrained on abnormality data such as a support vector machine model,another neural network model, or any supervised classification modeltrained on medical scans, or portions of medical scans, that includeknown abnormality classifying data to generate inference data for someor all of the classification categories. For example, the abnormalityclassification function can include another medical scan analysisfunction. Classification data 1375 in each of a plurality ofclassification categories can also be assigned their own calculatedconfidence score, which can also be generated by utilizing theabnormality classification function. Output to the abnormalityclassification function can also include at least one identified similarmedical scan and/or at least one identified similar cropped image, forexample, based on the training data. The abnormality classification stepcan also be included in the inference step 1354, where the inferredoutput vector or other inference data 1370 of the medical scan imageanalysis function includes the classification data 1375.

The abnormality classification function can be trained on full medicalscans and/or one or more cropped or full selected image slices frommedical scans that contain an abnormality. For example, the abnormalityclassification function can be trained on a set of two-dimensionalcropped slices that include abnormalities. The selected image slicesand/or the cropped region in each selected image slice for each scan inthe training set can be automatically selected based upon the knownlocation of the abnormality. Input to the abnormality classificationfunction can include the full medical scan, one or more selected fullimage slices, and/or one or more selected image slices cropped based ona selected region. Thus, the abnormality classification step can includeautomatically selecting one or more image slices that include thedetected abnormality. The slice selection can include selecting thecenter slice in a set of consecutive slices that are determined toinclude the abnormality or selecting a slice that has the largestcross-section of the abnormality, or selecting one or more slices basedon other criteria. The abnormality classification step can also includeautomatically generating one or more cropped two-dimensional imagescorresponding to the one or more of the selected image slices based onan automatically selected region that includes the abnormality.

Input to the abnormality classification function can also include otherdata associated with the medical scan, including patient history, riskfactors, or other metadata. The abnormality classification step can alsoinclude determining some or all of the characteristics based on data ofthe medical scan itself. For example, the abnormality size and volumecan be determined based on a number of pixels determined to be part ofthe detected abnormality. Other classifiers such as abnormality textureand/or smoothness can be determined by performing one or more otherpreprocessing functions on the image specifically designed tocharacterize such features. Such preprocessed characteristics can beincluded in the input to the abnormality classification function to themore difficult task of assigning a medical code or generating otherdiagnosis data. The training data can also be preprocessed to includesuch preprocessed features.

A similar scan identification step 1376 can also be performed on amedical scan with a detected abnormality and/or can be performed on theabnormality region 1373 determined in the detection step 1372. Thesimilar scan identification step 1376 can include generating similarabnormality data 1377, for example, by identifying one or more similarmedical scans or one or more similar cropped two-dimensional images froma database of medical scans and/or database of cropped two-dimensionalimages. Similar medical scans and/or cropped images can include medicalscans or cropped images that are visually similar, medical scans orcropped images that have known abnormalities in a similar location to aninferred abnormality location of the given medical scan, medical scansthat have known abnormalities with similar characteristics to inferredcharacteristics of an abnormality in the given scan, medical scans withsimilar patient history and/or similar risk factors, or some combinationof these factors and/or other known and/or inferred factors. The similarabnormality data 1377 can be mapped to the medical scan to populate someor all of its corresponding similar scan data 480 for use by one or moreother subsystems 101 and/or client devices 120.

The similar scans identification step 1376 can include performing a scansimilarity algorithm, which can include generating a feature vector forthe given medical scan and for medical scans in the set of medicalscans, where the feature vector can be generated based on quantitativeand/or category based visual features, inferred features, abnormalitylocation and/or characteristics such as the predetermined size and/orvolume, patient history and/or risk factor features, or other known orinferred features. A medical scan similarity analysis function can beapplied to the feature vector of the given medical scan and one or morefeature vectors of medical scans in the set. The medical scan similarityanalysis function can include computing a similarity distance such asthe Euclidian distance between the feature vectors, and assigning thesimilarity distance to the corresponding medical scan in the set.Similar medical scans can be identified based on determining one or moremedical scans in the set with a smallest computed similarity distance,based on ranking medical scans in the set based on the computedsimilarity distances and identifying a designated number of top rankedmedical scans, and/or based on determining if a similarity distancebetween the given medical scan and a medical scan in the set is smallerthan a similarity threshold. Similar medical scans can also beidentified based on determining medical scans in a database that mappedto a medical code that matches the medical code of the medical scan, ormapped to other matching classifying data. A set of identified similarmedical scans can also be filtered based on other inputted orautomatically generated criteria, where for example only medical scanswith reliable diagnosis data or rich patient reports, medical scans withcorresponding with longitudinal data in the patient file such asmultiple subsequent scans taken at later dates, medical scans withpatient data that corresponds to risk factors of the given patient, orother identified criteria, where only a subset of scans that comparefavorably to the criteria are selected from the set and/or only ahighest ranked single scan or subset of scans are selected from the set,where the ranking is automatically computed based on the criteria.Filtering the similar scans in this fashion can include calculating, orcan be based on previously calculated, one or more scores as discussedherein. For example, the ranking can be based on a longitudinal qualityscore, such as the longitudinal quality score 434, which can becalculated for an identified medical scan based on a number ofsubsequent and/or previous scans for the patient. Alternatively or inaddition, the ranking can be based on a confidence score associated withdiagnosis data of the scan, such as confidence score data 460, based onperformance score data 530 associated with a user or medical entityassociated with the scan, based on an amount of patient history data ordata in the medical scan entry 352, or other quality factors. Theidentified similar medical scans can be filtered based on ranking thescans based on their quality score and/or based on comparing theirquality score to a quality score threshold. In some embodiments, alongitudinal threshold must be reached, and only scans that comparefavorably to the longitudinal threshold will be selected. For example,only scans with at least three scans on file for the patient and finalbiopsy data will be included.

In some embodiments, the similarity algorithm can be utilized inaddition to or instead of the trained abnormality classificationfunction to determine some or all of the inferred classification data1375 of the medical scan, based on the classification data such asabnormality classification data 445 or other diagnosis data 440 mappedto one or more of the identified similar scans. In other embodiments,the similarity algorithm is merely used to identify similar scans forreview by medical professionals to aid in review, diagnosis, and/orgenerating medical reports for the medical image.

A display parameter step 1378 can be performed based on the detectionand/or classification of the abnormality. The display parameter step caninclude generating display parameter data 1379, which can includeparameters that can be used by an interactive interface to best displayeach abnormality. The same or different display parameters can begenerated for each abnormality. The display parameter data generated inthe display parameter step 1378 can be mapped to the medical scan topopulate some or all of its corresponding display parameter data 470 foruse by one or more other subsystems 101 and/or client devices 120.

Performing the display parameter step 1378 can include selecting one ormore image slices that include the abnormality by determining the one ormore image slices that include the abnormality and/or determining one ormore image slices that has a most optimal two-dimensional view of theabnormality, for example by selecting the center slice in a set ofconsecutive slices that are determined to include the abnormality,selecting a slice that has the largest cross-section of the abnormality,selecting a slice that includes a two-dimensional image of theabnormality that is most similar to a selected most similartwo-dimensional-image, selecting the slice that was used as input to theabnormality classification step and/or similar scan identification step,or based on other criteria. This can also include automatically croppingone or more selected image slices based on an identified region thatincludes the abnormality. This can also select an ideal Hounsfieldwindow that best displays the abnormality. This can also includeselecting other display parameters based on data generated by themedical scan interface evaluating system and based on the medical scan.

FIGS. 14A-14B present an embodiment of a medical scan natural languageanalysis system 114. The medical scan natural language analysis system114 can determine a training set of medical scans with medical codes,such as medical codes 447, determined to be truth data. Correspondingmedical reports, included in report data 449, and/or other naturallanguage text data associated with a medical scan, such as naturallanguage text data 448, can be utilized to train a medical scan naturallanguage analysis function by generating a medical report naturallanguage model. The medical scan natural language analysis function canbe utilized to generate inference data for incoming medical reports forother medical scans to automatically determine corresponding medicalcodes, which can be mapped to corresponding medical scans as medicalcodes 447. Medical codes 447 assigned to medical scans by utilizing themedical report natural language model can be utilized by othersubsystems, for example, to train other medical scan analysis functions,to be used as truth data to verify annotations provided via othersubsystems, to aid in diagnosis, or otherwise be used by othersubsystems as described herein.

In various embodiments, the medical scan natural language analysissystem 114 is operable to generate a medical report natural languagemodel based on a selected set of medical reports of a plurality ofmedical reports and the at least one medical code mapped to each of theselected set of medical reports. A medical report that is not includedin the selected set is received via a network. A medical code isdetermined by utilizing the medical report natural language model on thefirst medical report. The medical code is mapped to a medical scancorresponding to the medical report, for example, where the medical scanis assigned to medical code 447. In various embodiments, additionaldiagnosis data 440 is also generated by the medical report naturallanguage model and is mapped to the corresponding medical scan. Invarious embodiments, the medical scan natural language analysis systemcan generate and/or utilize the medical scan natural language analysisfunction as described herein in conjunction with generating andutilizing the medical report natural language model.

FIG. 14A presents a learning step 1405. A medical report training set1420 that includes the selected set of medical reports of report data449 and corresponding medical codes 447 can be retrieved from themedical scan database 342 by the medical scan natural language analysissystem via the network 150. A learning algorithm 1410 can utilizenatural language processing techniques to generate a medical reportnatural language model 1450 based on the medical report training set1420. The medical report natural language model 1450 can include or beused to generate the medical scan natural language analysis function.FIG. 14B presents a training set 1425. A new medical report 1449 can bereceived from the medical scan database 342 or from a client device 120via the network 150. The medical report natural language model 1450 canbe utilized to determine at least one new medical code 447 from aplurality of possible medical codes 447. This new medical code 447 canbe sent to a client device 120 and/or mapped to the report data 449and/or corresponding medical scan in the medical scan database 342.

The medical scan natural language analysis system 114 can be utilized inconjunction with the medical scan report labeling system 104, and thesystems can share access to the medical label alias database 920. Themedical scan natural language analysis function can be utilized by themedical scan report labeling system 104 when performing the medicalreport analysis function to generate medical codes for medical reportsautomatically, where the medical scan natural language analysis functionis trained on a set medical reports previously labeled and/or trained bythe medical label alias database 920 of alias mapping pairs 925. Some orall of the automatically generated medical codes can still be sent toexpert users for review, and performance score data 630 of medical scannatural language analysis function can be updated accordingly based onexpert review. Model remediation, such as remediation step 1140, can beperformed by the medical scan natural language analysis system 114 oranother subsystem such as the medical scan diagnosing system 108 whenthe performance score data 630 indicates that the medical scan naturallanguage analysis function needs to be retrained. The medical scannatural language analysis system 114 can also be used to generate newalias mapping pairs 925 for inclusion in the medical label aliasdatabase 920. The medical report natural language model can also betrained on medical reports corresponding to medical scans with medicalcodes 447 that have already been assigned in the medical scan databaseby other subsystems.

The medical report natural language model can be a fully convolutionalneural network or another neural network. Generating the medical reportnatural language model can be based on techniques described inconjunction with the medical scan image analysis system 112 and based onlearning algorithms that utilize natural language processing techniques.Generating the medical report natural language can include utilizing aforward propagation algorithm on the plurality of medical reports togenerate a preliminary set of neural network parameters, and can includeutilizing a back propagation algorithm to generate an updated set ofneural network parameters based on a calculated set of parameter errorsand the preliminary set of neural network parameters. Determiningmedical codes for new medical reports can include utilizing the forwardpropagation algorithm on the new medical reports based on the updatedset of neural network parameters.

Utilizing the medical report natural language model to determine thefirst medical code can include identifying a relevant medical term inthe first medical report. After processing the relevant medical term andthe medical code can be transmitted to a client device via the networkfor display by a display device in conjunction with the medical report.The relevant medical term is identified in the natural language textdata of the first medical report in conjunction with displaying thefirst medical code, for example, where the relevant medical term ishighlighted or otherwise indicated. Display of the relevant medical termcan be based on a corresponding interface feature, and can be presentedin conjunction with the medical scan assisted review system 102 and/orcan be presented to an expert user of the medical scan report labelingsystem 104. In various embodiments, the relevant medical term isassociated with an alias mapping pair 925 utilized to determine themedical code. In other embodiments, a user of the client device canelect to add the relevant medical term and the medical code as a newalias mapping pair 925 for the medical label alias database 920.

The medical scan natural language analysis system 114 can also beutilized to generate the medical report generating function. The trainedmedical report natural language model 1450 can be utilized to take imagedata 410 of medical scans, diagnosis data 440, or other data of amedical scan entry 352 as input and produce a written report as output.The medical report generating function can be trained on the same ordifferent training set as the medical scan natural language analysisfunction.

FIGS. 14C-14D provide examples of medical codes 447 determined byutilizing an embodiment of a medical scan natural language analysissystem 114 on a report data 449. Some or all of the text of report data449 and/or some or all medical codes 447 can be presented by aninteractive interface displayed on a client device, can be mapped tomedical scans in the medical scan database, and/or can be utilized byone or more additional subsystems. While the medical codes shown includeICD-9 codes and CPT codes determined based on the medical report, anymedical codes 447 described herein can be determined.

FIG. 15 presents an embodiment of a medical scan comparison system 116.Features of the medical scan comparison system 116 can be utilized byone or more subsystems to identify and/or display similar medical scans,for example, to perform or determine function parameters for the medicalscan similarity analysis function, to generate or retrieve similar scandata 1510, which can include some or all of similar scan data 480, orotherwise compare medical scan data. The medical scan comparison system116 can also utilize some or all features of other subsystems asdescribed herein.

As illustrated in FIG. 15, the medical scan comparison system 116 canreceive a medical scan via network 150. The medical scan comparisonsystem can determine at least one similar medical scan and generatesimilar scan data 1510 for transmission to a client device 120 fordisplay, for example, in conjunction with the medical scan assistedreview system 102.

In various embodiments, the medical scan comparison system 116 isoperable to receive a medical scan via a network and to generate similarscan data. The similar scan data includes a subset of medical scans froma medical scan database and is generated by performing an abnormalitysimilarity function, such as medical scan similarity analysis function,to determine that a set of abnormalities included in the subset ofmedical scans compare favorably to an abnormality identified in themedical scan. At least one cross-sectional image is selected from eachmedical scan of the subset of medical scans for display on a displaydevice associated with a user of the medical scan comparison system inconjunction with the medical scan.

FIG. 16 presents an embodiment of a method for execution by a medicalscan diagnosing system 108 or other subsystem as described herein thatincludes a processor. Step 1602 includes receiving, via a network, afirst medical scan. Step 1604 includes generating first diagnosis dataof the first medical scan by performing a first medical scan inferencefunction on the first medical scan. Step 1606 includes transmitting, viathe network, the first medical scan to a first client device associatedwith a first user of the medical scan diagnosing system in response tothe first diagnosis data indicating that the first medical scancorresponds to a non-normal diagnosis, where the first medical scan isdisplayed to the first user via an interactive interface displayed by afirst display device corresponding to the first client device. Step 1608includes receiving, via a network, first review data from the firstclient device, where the first review data is generated by the firstclient device in response to a first prompt displayed by the firstdisplay device via the interactive interface to provide the first reviewdata. Step 1610 includes generating updated first diagnosis data basedon the first review data. Step 1612 includes transmitting, via thenetwork, the updated first diagnosis data to a second client deviceassociated with a requesting entity.

In various embodiments, an updated first medical scan inference functionis generated in response to determining the first review data indicatesthat the first diagnosis data is incorrect. In various embodiments, thefirst diagnosis data is transmitted to the first client device, and thefirst diagnosis data is displayed to the first user via the interactiveinterface in conjunction with the first medical scan. In variousembodiments, the first review data includes second diagnosis data inresponse to the first prompt displayed by the first display device viathe interactive interface to provide the second diagnosis data.Determining the first review data indicates that the first diagnosisdata is incorrect is based on comparing the first diagnosis data to thesecond diagnosis data.

In various embodiments, a second medical scan is received. Seconddiagnosis data of the second medical scan is generated by performing thefirst medical scan inference function on the second medical scan. Modelquality check condition data is generated in response to the seconddiagnosis data indicating that the second medical scan corresponds to anormal diagnosis. The second medical scan is transmitted to the firstclient device in response to the model quality check condition dataindicating that a quality check condition is met. The second medicalscan is displayed to the first user via the interactive interfacedisplayed by a first display device corresponding to the first clientdevice. Second review data from the first client device, where thesecond review data is generated by the first client device in responseto a second prompt displayed by the first display device via theinteractive interface to provide the second review data. An updatedfirst medical scan inference function is generated in response todetermining the second review data indicates that the second diagnosisdata is incorrect. In various embodiments, generating the model qualitycheck condition data is based on determining if a quality check quota ismet.

In various embodiments, the first medical scan inference function isselected from a plurality of medical scan inference functions based on afirst medical scan classifier corresponding to the first medical scan.In various embodiments, a new medical scan classifier is generated byperforming a medical scan classifier function on the first medical scan.The first medical scan inference function is selected based on the newmedical scan classifier in response to determining that the firstmedical scan classifier compares unfavorably to the new medical scanclassifier. In various embodiments, the first medical scan classifier isindicated in metadata of the first medical scan, retrieved from amedical scan database. Updated metadata corresponding to the firstmedical scan is generated based on the new medical scan classifier. Theupdated metadata is mapped to the first medical scan in the medical scandatabase. In various embodiments, the first medical scan classifierindicates one of a plurality of anatomical regions, and the plurality ofmedical scan inference functions correspond to the plurality ofanatomical regions.

In various embodiments, a user database includes a plurality ofregistered entities that includes the requesting entity. The firstmedical scan is received from the second client device, and usage datacorresponding to the requesting entity is retrieved from the userdatabase in response to receiving the first medical scan. The firstdiagnosis data is generated in response to determining the usage datacompares favorably to a usage quota.

In various embodiments, test diagnosis data is generated for eachmedical scan in a medical scan test set by performing the first medicalscan inference function on each medical scan in the medical scan testset. Model quality check data is generated by comparing the testdiagnosis data to truth diagnosis data corresponding to the medical scantest set. The first medical scan inference function is updated inresponse to determining that the model quality check data comparesunfavorably to the truth diagnosis data. In various embodiments, thetest diagnosis data is generated in response to determining an update toa hardware component and/or a software component of the processingsystem and/or the memory.

FIG. 17 presents an embodiment of a method for execution by a medicalscan report labeling system 104 or other subsystem as described hereinthat includes a processor. Step 1702 include transmitting, via anetwork, a first medical report in a plurality of medical reports to afirst client device associated with a first user in a plurality of usersof the medical scan report labeling system, where natural language textdata of the first medical report is displayed to the first user via afirst interactive interface displayed by a first display devicecorresponding to the first client device. Step 1704 includes receiving,via the network, first identified medical condition term data from thefirst client device, where the first identified medical condition termdata is generated by the first client device in response to a firstprompt via the first interactive interface displayed by the firstdisplay device to identify a medical condition term based on the naturallanguage text data. Step 1706 includes identifying a first alias mappingpair of a plurality of alias mapping pairs in a medical label aliasdatabase, where each of the plurality of alias mapping pairs includesone of a plurality of medical condition terms and a corresponding one ofa plurality of medical codes, and where the first alias mapping pair isidentified by determining a first medical condition term of a pluralityof medical condition terms that corresponds to the first alias mappingpair compares favorably to the first identified medical condition termdata. Step 1708 includes retrieving a first medical code of theplurality of medical codes that corresponds to the first alias mappingpair from the medical label alias database in response to identifyingthe one of the plurality of medical condition terms. Step 1710 includesidentifying an expert user in the plurality of users. Step 1712 includestransmitting, via the network, the first medical code and a firstmedical scan of a plurality of medical scans that corresponds to thefirst medical report to a second client device associated with theexpert user, where the first medical code and the first medical scan aredisplayed to the expert user via a second interactive interfacedisplayed by a second display device corresponding to the second clientdevice. Step 1714 includes receiving, via the network, first accuracydata from the second client device, where the first accuracy data isgenerated by the second client device in response to a second promptdisplayed by the second display device via the second interactiveinterface to provide first accuracy data based on the first medical codeand the first medical scan. Step 1716 includes mapping the first medicalcode to the first medical scan in a medical scan database in response tothe first accuracy data indicating that the first medical code comparesfavorably to the first medical scan.

In various embodiments, the plurality of medical codes correspond toSNOMED codes, CPT codes, ICD-9 codes, and/or ICD-10 codes. In variousembodiments, a corrected medical code is received from the second clientdevice, where the corrected medical code is generated by the secondclient device in response to a third prompt displayed by the seconddisplay device via the second interactive interface to identify acorrected medical code, and where the third prompt is displayed inresponse to the first accuracy data indicating that the first medicalcode compares unfavorably to the first medical scan. The correctedmedical code to the first medical scan in the medical scan database. Invarious embodiments, the first medical report is transmitted to thesecond client device for display to the expert user via a secondinteractive interface. A new alias mapping pair is received from thesecond client device that includes the corrected medical code, where thenew alias mapping pair is generated by the second client device inresponse to a fourth prompt displayed by the second display device viathe second interactive interface to identify a new alias mapping pairbased on the first medical report and the first medical scan, and wherea new medical condition term of the new alias mapping pair is notincluded in the medical label alias database. The new alias mapping pairis added to the medical label alias database.

In various embodiments, the first identified medical condition term datais generated by the first client device based on an identified segmentof consecutive words in the natural language text data of the firstmedical report. In various embodiments, the first identified medicalcondition term data is generated by the first client device based on anidentified plurality of words in the natural language text data of thefirst medical report, where at least two of the identified plurality ofwords are not consecutive words in the natural language text data of thefirst medical report. In various embodiments, the first identifiedmedical condition term data is generated by the first client devicebased on a plurality of words, where at least one word in the pluralityof words is not included in the natural language text data of the firstmedical report, and where the at least one word is based on keyboardand/or voice input via the first interactive interface. In variousembodiments, the first medical condition term of the first alias mappingpair and the first identified medical condition term data received fromthe first client device differ by at least one word. Determining thatfirst medical condition term compares favorably to the first identifiedmedical condition term data includes calculating a similarity scorebetween the first medical condition term and the first identifiedmedical condition term data and further includes determining that thesimilarity score compares favorably to a similarity threshold.

In various embodiments, a second medical report in the plurality ofmedical reports is transmitted to a third client device associated witha third user in a plurality of users of the medical scan report labelingsystem. The natural language text data of the second medical report isdisplayed to the third user via a third interactive interface displayedby a third display device corresponding to the third client device.Second identified medical condition term data is received from the thirdclient device, where the second identified medical condition term datais generated by the third client device in response to a third promptvia the third interactive interface displayed by the third displaydevice to identify a medical condition term based on the naturallanguage text data. The second medical report and a second medical scanof the plurality of medical scans that corresponds to the second medicalreport are transmitted to the second client device associated with theexpert user in response to determining that the medical label aliasdatabase does not include a one of the plurality of medical conditionterms that compares favorably to the second identified medical conditionterm data. The natural language text data of the second medical reportand the one of the plurality of medical scans are displayed to theexpert user via a second interactive interface displayed by a seconddisplay device corresponding to the second client device. A new aliasmapping pair is received from the second client device. The new aliasmapping pair is generated by the second client device in response to afourth prompt displayed by the second display device via the secondinteractive interface to identify a new alias mapping pair based on thesecond medical report and the second medical scan, where a new medicalcondition term of the new alias mapping pair is not included in themedical label alias database. The new alias mapping pair is added to themedical label alias database, and a second medical code in the new aliasmapping pair is mapped to the second medical scan.

In various embodiments, at least one of the plurality of medical codesis included in more than one of the plurality of alias mapping pairs. Invarious embodiments, the first medical report and the first medical codeare added to a training set that includes a subset of the plurality ofmedical reports and a corresponding plurality of medical codes. Amedical report natural language model is generated based on the trainingset. A second medical report in the plurality of medical reports that isnot included in the training set is identified. A second medical code ofthe plurality of medical codes is determined by utilizing the medicalreport natural language model on the second medical report, and thesecond medical code is mapped to a second medical scan corresponding tothe second medical report. In various embodiments, the second medicalcode, as well as the second medical report and/or second medical scan inthe plurality of medical scans that corresponds to the second medicalreport, are transmitted to the second client device. The second medicalcode, as well as the second medical report and/or the second medicalscan, are displayed to the expert user via a third interactive interfacedisplayed by the second display device corresponding to the secondclient device. Model accuracy data is received from the second clientdevice, where the model accuracy data is generated by the second clientdevice in response to a third prompt displayed by the second displaydevice via the third interactive interface to provide model accuracydata based on the first medical code and the second medical reportand/or the second medical scan. The second medical code is mapped to thesecond medical scan in the medical scan database in response to themodel accuracy data indicating that the second medical code comparesfavorably to the second medical report and/or the second medical scan.

FIG. 18 presents an embodiment of a method for execution by a medicalscan interface feature evaluator system 110 or other subsystem asdescribed herein that includes a processor. Step 1802 includes selectinga set of user interface features for evaluation. Step 1804 includesselecting a first set of medical scans from a medical scan database.Step 1806 includes generating a first ordered image-to-prompt mapping byselecting one of the set of user interface features to be displayed witheach of the medical scans in the first set of medical scans and byselecting a first order of the first set of medical scans. Step 1808includes transmitting, via a network, the first set of medical scans andthe first ordered image-to-prompt mapping to a first set of clientdevices corresponding to a first set of users selected from a pluralityof users. Step 1810 includes receiving, via the network, a first set ofresponses from each of the first set of client devices, where the firstset of responses is generated by each of the first set of client devicein response to sequentially displaying, based on the first order, eachof the first set of medical scans in conjunction with a mapped one ofthe set of user interface features indicated in the first orderedimage-to-prompt mapping via a user interface on a display deviceassociated with the each of the first set of client devices. Step 1812includes generating first response score data for each response of thefirst set of responses received from each of the first set of clientdevices by comparing each response to truth annotation data of acorresponding medical scan of the first set of medical scans indicatedby the first ordered image-to-prompt mapping. Step 1814 includesgenerating interface feature score data corresponding to each userinterface feature in the set of user interface features based onaggregating the first response score data for each response of each ofthe first set of responses. Step 1816 includes generating a ranking ofthe set of user interface features based on the interface feature scoredata.

In various embodiments, a lowest ranked user interface feature in theranking is automatically removed from the set of user interfacefeatures. In various embodiments, at least one response in the first setof responses indicates that a corresponding medical scan is normal inresponse to one of the first set of client devices displaying one of theset of user interface features that includes a prompt to select eitherthat a displayed medical scan is normal or that the displayed medicalscan includes an abnormality. Generating the first response score datafor the at least one response includes assigning a low score to the oneresponse based on determining that the truth annotation data of an atleast one corresponding medical scan indicates at least one abnormality.In various embodiments, at least one response in the first set ofresponses includes abnormality classification data generated by one ofthe first set of client devices in response to displaying one of the setof user interface features that includes a prompt to classify anabnormality. Generating the first response score data of the at leastone response includes comparing the abnormality classification data ofthe at least one response to the truth annotation data of an at leastone corresponding medical scan. In various embodiments, at least oneresponse in the first set of responses includes an identified regiongenerated by one of the first set of client devices in response todisplaying one of the set of user interface features that includes aprompt to identify a region of the displayed medical scan that includesan abnormality. Generating the first response score data includescomparing the identified region of the at least one response to thetruth annotation data of an at least one corresponding medical scan.

In various embodiments, a second set of medical scans is selected fromthe medical scan database. A second ordered image-to-prompt mapping isgenerated by selecting one of the set of user interface features to bedisplayed with each of the medical scans in the second set of medicalscans and by selecting a second order of the second set of medicalscans, where the second ordered image-to-prompt mapping is differentthan the first ordered image-to-prompt mapping. The second set ofmedical scans and the set of user interface features are transmitted toa second client device corresponding to a second user selected from theplurality of users, where the second client device is not included inthe first set of client devices. A second set of responses are receivedfrom the second client device, where the second set of responses isgenerated by the second client device in response to utilizing thesecond ordered image-to-prompt mapping to sequentially display each ofthe second set of medical scans in conjunction with a user interfacebased on the mapped one of the set of user interface features on asecond user interface on a second display device associated with secondclient device. Second response score data is generated for each responseof each of the second set of responses by comparing each of the secondset of responses to the truth annotation data of a corresponding medicalscan indicated by the second ordered image-to-prompt mapping. Generatingthe interface feature score data corresponding to each user interfacefeature in the set of user interface features is further based on thesecond response score data. In various embodiments, the first orderedimage-to-prompt mapping is generated based on first performance scoredata of the first set of users, and the second ordered image-to-promptmapping is generated based on second performance score data of thesecond user.

In various embodiments, a new medical scan is received via the network.A new interface mapping is generated for the new medical scan byselecting at least one user interface feature from the set of userinterface features based on the ranking of the set of user interfacefeatures. The new medical scan and the new interface mapping aretransmitted to a third client device associated with a third userselected from the plurality of users. Third response data is receivedfrom the third client device, where the third response data is generatedby the third client device in response to utilizing the new interfacemapping to display the new medical scan in conjunction with the selectedat least one user interface feature a third user interface on a thirddisplay device associated with the one of the third client device. Newannotation data is generated based on the third response data, and thenew annotation data is mapped to the new medical scan in the medicalscan database. In various embodiments, the at least one user interfacefeature is selected based by determining at least one highest rankeduser interface feature in the ranking. In various embodiments, the firstset of medical scans each correspond to one of a set of scan categories.The ranking includes a set of categorized rankings corresponding to theset of scan categories, and the at least one user interface feature isselected for the new interface mapping based on a one of the set ofcategorized rankings that corresponds to a scan category of the newmedical scan.

FIG. 19 presents an embodiment of a method for execution by a medicalscan image analysis system 112 or other subsystem as described hereinthat includes a processor. Step 1902 includes receiving a plurality ofmedical scans via a network, where each of the plurality of medicalscans represents a three-dimensional anatomical region and includes aplurality of cross-sectional image slices. Step 1904 includes generatinga plurality of three-dimensional subregions corresponding to each of theplurality of medical scans by selecting a proper subset of the pluralityof cross-sectional image slices from each of the plurality of medicalscans, and by further selecting a two-dimensional subregion from eachproper subset of the plurality of cross-sectional image slices of theeach of the medical scans. Step 1906 includes performing a learningalgorithm on the plurality of three-dimensional subregions to generate afully convolutional neural network. Step 1908 includes receiving a newmedical scan that is not included in the plurality of medical scans.Step 1910 includes generating inference data corresponding to the newmedical scan by performing a first inference algorithm on the newmedical scan by utilizing the fully convolutional neural network. Step1912 includes identifying an inferred abnormality in the new medicalscan based on the inference data.

In various embodiments, performing the learning algorithm includesutilizing a forward propagation algorithm on the plurality ofthree-dimensional subregions to generate a preliminary set of neuralnetwork parameters, and by utilizing a back propagation algorithm togenerate an updated set of neural network parameters based on acalculated set of parameter errors and the preliminary set of neuralnetwork parameters. Performing the first inference algorithm includesutilizing the forward propagation algorithm on the new medical scanbased on the updated set of neural network parameters.

In various embodiments, at least one of the plurality of medical scansincludes a known abnormality. Selecting the proper subset of theplurality of cross-sectional image slices of the at least one of theplurality of medical scans is based on a known location of the knownabnormality, and selecting the two-dimensional subregion from eachproper subset of the plurality of cross-sectional image slices is basedon the known location of the known abnormality. In various embodiments,each of the plurality of three-dimensional subregions of the at leastone of the plurality of medical scans include the known abnormality. Invarious embodiments, the each of the plurality of three-dimensionalsubregions are selected randomly or pseudo-randomly. A first subregionoption corresponding to a one of the plurality of medical scans has afirst selection probability based on a first proximity of the firstsubregion option to the known location of the known abnormality, and asecond subregion option corresponding to the one of the plurality ofmedical scans has a second selection probability based on a secondproximity of the first subregion option to the known location of theknown abnormality. The first selection probability is higher than thesecond selection probability based on the first proximity being nearerthan the second proximity.

In various embodiments, a first number of pixels in a onethree-dimensional subregions of the plurality of three-dimensionalsubregions is smaller than a second number of pixels in a correspondingone of the plurality of medical scans. In various embodiments,generating the inference data includes generating a plurality ofabnormality probabilities. Each of the plurality of abnormalityprobabilities is mapped to a pixel location of each of a plurality ofcross-sectional image slices of the new medical scan, and identifyingthe inferred abnormality is based on the plurality of abnormalityprobabilities. In various embodiments, classification data correspondingto the inferred abnormality is generated based on the inference data. Invarious embodiments, generating the classification data includesgenerating a new three-dimensional subregion of the new medical scanthat includes the inferred abnormality and by performing a secondinference algorithm on the new three-dimensional subregion.

In various embodiments, a density window for the plurality of medicalscans is determined by determining a low end density cut-off value andhigh end density cut-off value based on Hounsfield units of the each ofthe plurality of medical scans. A plurality of preprocessed medicalscans are generated by utilizing the density window to mask a firstsubset of pixels of the plurality of medical scans. The plurality ofthree-dimensional subregions are generated by utilizing the plurality ofpreprocessed medical scans. In various embodiments, an updated low enddensity cut-off value and an updated high end density cut-off value aregenerated in a first iteration of the learning algorithm. A plurality ofprocessed medical scans are generated by utilizing the updated low enddensity cut-off value and the updated high end density cut-off value tomask a second subset of pixels of the plurality of medical scans. Asubsequent iteration of the learning algorithm is performed on theplurality of processed medical scans to generate the fully convolutionalneural network.

In various embodiments, padded data is generated for each of theplurality of three-dimensional subregions based on a data reflection ata plurality of boundaries of each of the plurality of three-dimensionalsubregions. Performing the learning algorithm includes convolving eachof the plurality of three-dimensional subregions, where the padded datais utilized to convolve each of the plurality of three-dimensionalsubregions at the plurality of boundaries.

FIG. 20 presents an embodiment of a method for execution by a medicalscan annotator system 106 or other subsystem as described herein thatincludes a processor. Step 2002 includes selecting a first medical scanfrom a medical scan database for transmission via a network to a firstclient device associated with a first user in a user database, where thefirst medical scan is displayed to the first user via an interactiveinterface displayed by a first display device corresponding to the firstclient device. Step 2004 includes receiving first annotation data fromthe first client device via the network, where the first annotation datais generated by the first client device in response to a prompt via theinteractive interface displayed by the first display device to providethe first annotation data corresponding to the first medical scan. Step2006 includes transmitting the first medical scan to a second clientdevice associated with a second user in the user database via thenetwork, where the first medical scan is displayed to the second uservia the interactive interface displayed by a second display devicecorresponding to the second client device. Step 2008 includes receivingsecond annotation data from the second client device via the network,where the second annotation data is generated by the second clientdevice in response to a prompt via the interactive interface displayedby the second display device to provide the second annotation datacorresponding to the first medical scan. Step 2010 includes generatingannotation similarity data by comparing the first annotation data to thesecond annotation data. Step 2012 includes generating consensusannotation data based on the first annotation data and the secondannotation data in response to the annotation similarity data indicatingthat a difference between the first annotation data and the secondannotation data compares favorably to an annotation discrepancythreshold. Step 2014 includes mapping the consensus annotation data tothe first medical scan in the medical scan database.

In various embodiments, a first expert user is selected from a set ofexpert users in the user database in response to the annotationsimilarity data indicating that the difference between the firstannotation data and the second annotation data compares unfavorably tothe annotation discrepancy threshold. The medical scan is transmitted toa third client device associated with the first expert user. The firstmedical scan is displayed to the first expert user via the interactiveinterface displayed by a third display device corresponding to the thirdclient device. Third annotation data is received from the third clientdevice. The third annotation data is generated by the third clientdevice in response to a first prompt via the interactive interfacedisplayed by the third display device to provide third annotation data.The third annotation data is mapped to the first medical scan in themedical scan database. In various embodiments, the first annotation dataand the second annotation data is transmitted to the third clientdevice, and the first annotation data and the second annotation data aredisplayed via the interactive interface. In various embodiments, theinteractive interface includes a prompt to select the first annotationdata or the second annotation data. The third annotation datacorresponds to the first annotation data in response to a selection ofthe first annotation data via the interactive interface. In variousembodiments, the user database stores user performance datacorresponding to the plurality of users. An annotation accuracy score isgenerated by comparing the first annotation data to the third annotationdata. The user performance data corresponding to the first user isupdated in the user database based on the annotation accuracy score.

In various embodiments, the medical scan database includes a set oftriaged medical scans and a set of corresponding priority valuesreceived from a triaging entity via the network. The first medical scanis selected from the set of triaged medical scans based on a ranking ofthe set of corresponding priority values, and the third annotation datais transmitted to the triaging entity via the network. The user databasestores user efficiency data corresponding to the plurality of users,where the set of triaged medical scans are assigned to a subset of theplurality of users based on the set of corresponding priority values andfurther based on the user efficiency data. In various embodiments, theuser database stores user specialization data corresponding to theplurality of users. A medical scan category corresponding to the firstmedical scan is determined, and the first user is selected based ondetermining that the user specialization data compares favorably to themedical scan category.

In various embodiments, the user database stores user performance datacorresponding to the plurality of users. A second medical scan isselected from the medical scan database for transmission via a networkto a fourth client device associated with a fourth user of the medicalscan annotator system, where the second medical scan is displayed to thefourth user via an interactive interface displayed by a fourth displaydevice corresponding to the fourth client device. Fourth annotation datais received from the fourth client device, where the fourth annotationdata is generated by the fourth client device based on user input to theinteractive interface in response to a prompt via the interactiveinterface displayed by the fourth display device to provide the fourthannotation data corresponding to the second medical scan. Truthannotation data mapped to the second medical scan is retrieved from amedical scan database. An annotation accuracy score is generated bycomparing the fourth annotation data to the truth annotation data. Userperformance data corresponding to the fourth user in the user databaseis updated based on the annotation accuracy score. In variousembodiments, truth annotation data is received from a fifth clientdevice corresponding to a second expert user. The truth annotation datato the second medical scan. In various embodiments, the truth annotationdata is generated by performing an automated annotating function on thesecond medical scan that utilizes a computer vision model, where thecomputer vision model is trained on a subset of the plurality of medicalscans. The truth annotation data is mapped to the second medical scan.

In various embodiments, the user performance data includesspecialization data that includes a plurality of category specialtyscores. At least one image category corresponding to the second medicalscan is determined. At least one category specialty score of thespecialization data of the fourth user is updated based on theannotation accuracy score, where the at least one category specialtyscore corresponds to the at least one image category. The fourth user isadded to a set of expert users in the user database in response todetermining that the updated user performance data compares favorably toan expert status threshold.

In various embodiments, the first annotation data indicates either thatthe first medical scan includes no abnormalities or that the firstmedical scan includes an abnormality. In various embodiments, theinteractive interface prompts the first user to provide abnormalityclassification data in response to the first annotation data indicatingthat the first medical scan includes an abnormality, and the firstannotation data further indicates the abnormality classification data.In various embodiments, the interactive interface prompts the first userto provide abnormality location data in response to the first annotationdata indicating that the first medical scan includes an abnormality, andthe first annotation data indicates the abnormality location data. Invarious embodiments, the first medical scan is a chest x-ray or a chestCT scan.

FIG. 21 presents an embodiment of a method for execution by a chestx-ray differential diagnosis system or other subsystem as describedherein that includes a processor. Step 2102 includes receiving aplurality of chest x-rays via a network. Step 2104 includes generatingabnormality pattern data for each of the chest x-rays by identifying atleast one pattern in the chest x-rays corresponding to an abnormality byutilizing a computer vision model, where the computer vision model istrained on a plurality of training chest x-rays. Step 2106 includesgenerating differential diagnosis data for each of the plurality ofchest x-rays based on the abnormality pattern data, Step 2108 includesreceiving, via the network, filtering parameters in response to a firstprompt via an interactive interface displayed on a display deviceassociated with a user with a user of the chest x-ray differentialdiagnosis system. Step 2110 includes generating a filtered chest x-rayqueue for transmission to a client device associated with the user fordisplay via the interactive interface, where the filtered chest x-rayqueue includes a proper subset of chest x-rays selected from theplurality of chest x-rays based on the filtering parameters and thedifferential diagnosis data. Step 2112 includes receiving, via thenetwork, chest x-ray selection data indicating one of the proper subsetof chest x-rays in response to a second prompt via the interactiveinterface. Step 2114 includes transmitting the differential diagnosisdata to the client device via the network corresponding to one of theplurality of chest x-rays indicated in the chest x-ray selection datafor display via the display device in conjunction with the one of theplurality of chest x-rays.

In various embodiments, the plurality of chest x-rays includes a normalsubset of chest x-rays with corresponding differential diagnosis dataindicating a normal diagnosis, and where the proper subset of chestx-rays is selected to automatically omit chest x-rays in the normalsubset of chest x-rays. In various embodiments, the differentialdiagnosis data for each of the plurality of chest x-rays includes aplurality of binary values indicating whether each of a plurality ofabnormality pattern types are present or not present based on theabnormality pattern data. In various embodiments, the abnormalitypattern data includes confidence score data corresponding to each of theplurality of abnormality pattern types. Generating the differentialdiagnosis data includes comparing the confidence score data for each ofthe plurality of abnormality pattern types to a first confidence scorethreshold. A first one of the plurality of binary values indicates acorresponding first one of the plurality of abnormality pattern types ispresent when the corresponding confidence score data compares favorablyto the first confidence score threshold, and a second one of theplurality of binary values indicates a corresponding second one of theplurality of abnormality pattern types is present when the correspondingconfidence score data compares unfavorably to the first confidence scorethreshold.

In various embodiments, a new confidence score threshold is received inresponse to a third prompt displayed via the interactive interface.Updated differential diagnosis data is generated by applying the newconfidence score threshold. An updated filtered chest x-ray queue isgenerated for transmission to the client device for display via theinteractive interface by generating a new proper subset of chest x-raysbased on the updated differential diagnosis data. In variousembodiments, the filtering parameters indicate a subset of the pluralityof abnormality pattern types, and generating the filtered chest x-rayqueue includes selecting ones of the plurality of chest x-rays withdifferential diagnosis data that indicates that at least one of thesubset of the plurality of abnormality pattern types in the filteringparameters is present to be included in the proper subset of chestx-rays. In various embodiments, the plurality of abnormality patterntypes includes at least one of: cardiomegaly, consolidation, effusion,emphysema, or fracture. In various embodiments, the plurality of chestx-rays are ordered based on priority data.

In various embodiments, corrected diagnosis data is received in responseto a third prompt via the interactive interface in conjunction with thedisplay of the one of the plurality of chest x-rays and the differentialdiagnosis data. The corrected diagnosis data is mapped to the one of theplurality of chest x-rays in a chest x-ray database. In variousembodiments, the plurality of chest x-rays are mapped to known diagnosisdata in a chest x-ray database. A similarity score is generated bycomparing the known diagnosis data to the differential diagnosis data.Accuracy data is generated by comparing the similarity score to anaccuracy threshold, where the accuracy data indicates that a discrepancyis present when the similarity score compares unfavorably to theaccuracy threshold, and where the accuracy data indicates that nodiscrepancy is present when the similarity score compares favorably tothe accuracy threshold. Generating the filtered chest x-ray queueincludes selecting only chest x-rays for the proper subset that havecorresponding accuracy data indicating that a discrepancy is present.

FIG. 22 presents an embodiment of a method for execution by a lungscreening assessment system or other subsystem as described herein thatincludes a processor. Step 2202 includes receiving a first chestcomputed tomography (CT) scan that includes a plurality of crosssectional images. Step 2204 includes generating nodule classificationdata of the first chest CT scan by utilizing a computer vision model toidentify a first nodule in the plurality of cross sectional images anddetermine an assessment score, where the computer vision model istrained on a plurality of training chest CT scans. Step 2206 includesgenerating a lung screening report for display on a display deviceassociated with a user of the lung screening assessment system, wherethe lung screening report includes the assessment score of the noduleclassification data.

In various embodiments, generating the nodule classification dataincludes calculating at least one of: a Fleischner score or a Lung-RADSscore, where the assessment score indicates the at least one of: theFleischner score or the Lung-RADS score. In various embodiments,generating the nodule classification data includes generating a firstsubset of chest CT scans from the plurality of training chest CT scansby applying a similarity function to determine that a set of nodulesincluded in the first subset of chest CT scans compare favorably to thefirst nodule. At least one cross sectional image is selected from eachchest CT scan of the first subset of chest CT scans for display on thedisplay device in conjunction with the lung screening report.

In various embodiments, a set of patient records corresponding to thefirst subset of chest CT scans are retrieved, and the lung screeningreport includes data from at least one patient record in the set ofpatient records. In various embodiments, first patient risk factor datacorresponding to the chest CT scan is retrieved, and patient risk factordata corresponding to each of the set of patient records is retrieved. Asubset of patient records is selected by identifying correspondingpatient risk factor data that compares favorably to the first patientrisk factor data, and the lung screening report includes only data frompatient records in the subset of patient records.

In various embodiments, each of the set of patient records includes aplurality of record entries and a corresponding plurality of dates. Asubset of the plurality of dates are identified in the each of the setof patient records that are more recent than a date associated with theone of the plurality of chest CT scans corresponding to the each of theset of patient records. A subset of the plurality of record entries ofthe each of the set of patient records that correspond to the subset ofthe plurality of dates are identified. Longitudinal data correspondingto each of the set of patient records is generated based on eachcorresponding subset of the plurality of record entries, and the lungscreening report includes the longitudinal data. In various embodiments,diagnosis prediction data corresponding to the first chest CT scan isgenerated based on the longitudinal data, and the lung screening reportincludes the diagnosis prediction data.

In various embodiments, generating the longitudinal data includescalculating a set of longitudinal quality scores corresponding to eachpatient record in the set of patient records. A ranking of the set ofpatient records is generated based on the set of longitudinal qualityscores. At least one patient record is removed from the set of patientrecords that corresponds to at least one lowest ranking, and the lungscreening report includes only data from remaining patient records inthe set of patient records. In various embodiments, calculating the setof longitudinal quality scores is based on a number of subsequent chestCT scans included in the each of the set of patient records. In variousembodiments, calculating the set of longitudinal quality scores is basedon determining a duration of time between a first scan date of eachchest CT scan in the first subset of chest CT scans and a second scandate of a most recent subsequent chest CT scan included in thecorresponding patient record in the set of patient records. In variousembodiments, a first longitudinal quality score in the set oflongitudinal quality scores corresponds to a first patient record, and asecond longitudinal quality score in the set of longitudinal qualityscores corresponds to a second patient record. The first longitudinalquality score is more favorable than a second longitudinal quality scorein response to determining that the first patient record includes biopsydata and the second patient record does not include biopsy data.

FIG. 23 presents an embodiment of a method for execution by a lungscreening assessment system or other subsystem as described herein thatincludes a processor. Step 2302 includes receiving a first medical scanvia a network. Step 2304 include generating similar scan data thatincludes a subset of medical scans from a medical scan database thatincludes a plurality of medical scans, where generating the similar scandata includes performing an abnormality similarity function to determinethat a set of abnormalities included in the subset of medical scanscompare favorably to an abnormality identified in the first medicalscan. Step 2306 includes transmitting at least one selectedcross-sectional image from each medical scan of the subset of medicalscans to a client device for display on a display device associated witha user of the medical scan comparison system in conjunction with thefirst medical scan.

In various embodiments, first patient risk factor data corresponding tothe first medical scan is retrieved from the medical scan database. Aset of patient risk factor data corresponding to the subset of medicalscans are retrieved from the medical scan database. A subset of patientrisk factor data in the set of patient risk factor data that comparesunfavorably to the first patient risk factor data is identified. Anupdated subset of medical scans is generated by removing medical scansthat correspond to the subset of patient risk factor data, and the atleast one selected cross-sectional image displayed by the display deviceare selected from each medical scan of the updated subset of medicalscans.

In various embodiments, a set of patient records corresponding to thesubset of medical scans is retrieved from the medical scan database.Medical scan report data is generated for display on the display devicethat includes data from at least one patient record in the set ofpatient records. In various embodiments, each of the set of patientrecords includes a plurality of record entries and a correspondingplurality of dates. A subset of the plurality of dates is identified inthe each of the set of patient records that are more recent than a dateassociated with the one of the plurality of medical scans correspondingto the each of the set of patient records. A subset of the plurality ofrecord entries of the each of the set of patient records that correspondto the subset of the plurality of dates is identified. Longitudinal datacorresponding to each of the set of patient records based on eachcorresponding subset of the plurality of record entries is generated,and the medical scan report data includes the longitudinal data. Invarious embodiments, diagnosis prediction data corresponding to thefirst medical scan is generated based on the longitudinal data, and themedical scan report data includes the diagnosis prediction data.

In various embodiments, a set of longitudinal quality scorescorresponding to the longitudinal data of each patient record in the setof patient records is calculated. A ranking of the set of patientrecords is generated based on the set of longitudinal quality scores. Atleast one patient record is removed from the set of patient records thatcorresponds to at least one lowest ranking. An updated subset of medicalscans is generated by removing at least one medical scan correspondingto the at least one patient record. The medical scan report dataincludes only data from remaining patient records in the set of patientrecords, and the at least one selected cross-sectional image displayedby the display device are selected from each medical scan of the updatedsubset of medical scans.

In various embodiments, calculating the set of longitudinal qualityscores is based on the number of subsequent medical scans included inthe each of the set of patient records. In various embodiments,calculating the set of longitudinal quality scores is based ondetermining a duration of time between a first scan date of each medicalscan in the subset of medical scans and a second scan date of a mostrecent subsequent medical scan included in the corresponding patientrecord in the set of patient records. In various embodiments, a firstlongitudinal quality score in the set of longitudinal quality scorescorresponds to a first patient record, and a second longitudinal qualityscore in the set of longitudinal quality scores corresponds to a secondpatient record. The first longitudinal quality score is more favorablethan a second longitudinal quality score in response to determining thatthe first patient record includes biopsy data and the second patientrecord does not include biopsy data.

In various embodiments, performing the abnormality similarity functionincludes comparing at least one cropped image slice of the first medicalscan to a plurality of cropped images slices of the plurality of medicalscans by applying computer vision techniques. At least one of theplurality of cropped image slices that compares favorably to asimilarity threshold is identified, and the subset of medical scanscorrespond to the at least one of the plurality of cropped image slicesthat compare favorably to the similarity threshold. In variousembodiments, the medical scan database includes a plurality of knownabnormality classification data mapped to the plurality of medicalscans. First abnormality classification data is determined for theabnormality of the first medical scan. A subset of known abnormalityclassification data of the plurality of known abnormality classificationdata that compares favorably to the first abnormality classificationdata is identified. Medical scans mapped to the subset of knownabnormality classification data are included in the subset of medicalscans.

FIG. 24 presents an embodiment of a method for execution by a medicalscan assisted review system 102 or other subsystem as described hereinthat includes a processor. Step 2402 includes receiving, via a network,a first medical scan for review. Step 2404 includes generatingabnormality data by identifying a plurality of abnormalities in thefirst medical scan by utilizing a computer vision model, where thecomputer vision model is trained on a plurality of training medicalscans, and where the abnormality data includes location data andclassification data for each of the plurality of abnormalities. Step2406 includes generating text describing each of the plurality ofabnormalities based on the abnormality data. Step 2408 includestransmitting the abnormality data and the text to a client deviceassociated with a user of the medical scan assisted review system, wherea display device associated with the client device displays theabnormality data in conjunction with the first medical scan via aninteractive interface, and where the display device further displays thetext via the interactive interface.

In various embodiments, display device displays only abnormality datacorresponding to one of the plurality of abnormalities in conjunctionwith the first medical scan in response to a first prompt via theinteractive interface to select the one of the plurality ofabnormalities for review. In various embodiments, a shape that surroundsthe one of the plurality of abnormalities in the first medical scan isdisplayed via the interactive interface in response to the first prompt.In various embodiments, the first medical scan includes a plurality ofimage slices, and the interactive interface automatically jumps to aselected one of the plurality of image slices that includes the one ofthe plurality of abnormalities in response to the first prompt.

In various embodiments, the display device displays the first medicalscan in a first view of the interactive interface. A set of similarmedical scans are selected from the plurality of training medical scansby applying a similarity function to determine that a known abnormalityincluded in each of the set of similar medical scan compares favorablyto a corresponding one of the plurality of abnormalities. The set ofsimilar medical scans are transmitted to the client device. One of theset of similar medical scans corresponding to the one of the pluralityof abnormalities is automatically displayed in a second view of theinteractive interface that is adjacent to the first view in response toa second prompt via the interactive interface.

In various embodiments, the first medical scan includes a firstplurality of image slices. The one of the set of similar medical scansincludes a second plurality of image slices. The second view of theinteractive interface automatically jumps to a one of the secondplurality of image slices in response to the second prompt thatcorresponds to a currently displayed one of the first plurality of imageslices of the first view. In various embodiments, the one of the secondplurality of image slices is selected based on determining that the oneof the second plurality of image slices compares favorably to a crosssectional anatomical region of the currently displayed one of the firstplurality of image slices of the first view. In various embodiments, theone of the second plurality of image slices is selected based ondetermining that the one of the second plurality of image slicesincludes a view of the known abnormality that compares favorably to theone of the plurality of abnormalities. In various embodiments, the firstview scrolls to a new one of the first plurality of image slices inresponse to a third prompt to scroll to the new one of the firstplurality of image slices via the interactive interface, and the secondview simultaneously scrolls to a corresponding new one of the secondplurality of image slices in response to the third prompt.

In various embodiments, each of the set of similar medical scans isassociated with a patient record, and each of the set of similar medicalscans are selected based on the associated patient record comparingfavorably to a longitudinal quality threshold. In various embodiments, amore recent medical scan indicated in the patient record associated withthe one of the set of similar medical scans corresponding to the one ofthe plurality of abnormalities is transmitted, where the more recentmedical scan is displayed by display device in response to a thirdprompt via the interactive interface. New abnormality data is receivedvia the network in response a second prompt via the interactiveinterface. A location of the new abnormality data is based on a regionidentified in the medical scan via the interactive interface. Updatedabnormality data is generated based on the new abnormality data. Invarious embodiments, edited text data is received via the network inresponse to a second prompt via the interactive interface. Updatedabnormality data and updated natural language text are generated basedon the edited text data.

FIG. 25 presents an embodiment of a method for execution by a medicalscan natural language analysis system or other subsystem as describedherein that includes a processor. Step 2502 includes generating amedical report natural language model based on a selected set of medicalreports of the plurality of medical reports and the at least one medicalcode mapped to each of the selected set of medical reports. Step 2504includes receiving a first medical report that is not included in theselected set via a network. Step 2506 includes determining a firstmedical code by utilizing the medical report natural language model onthe first medical report. Step 2508 includes mapping the first medicalcode of the plurality of medical codes to a first medical scancorresponding to the first medical report.

In various embodiments, generating the medical report natural languagemodel is further based on a plurality of alias mapping pairs. Each ofthe plurality of alias mapping pairs includes a one of a plurality ofmedical condition terms and a corresponding one of a plurality ofmedical codes. Each of the plurality of medical condition terms in theplurality of alias mapping pairs are unique, and each of the pluralityof medical condition terms includes at least one word. In variousembodiments, a new alias mapping pair that is not included in theplurality of alias mapping pairs is received; and an updated medicalreport natural model is generated based on the new alias mapping pair.

In various embodiments, generating the medical report natural languagemodel includes generating a plurality of alias mapping pairs, where eachof plurality of alias mapping pairs includes a one of a plurality ofmedical condition terms and a corresponding one of a plurality ofmedical codes. Each of the plurality of medical condition terms in theplurality of alias mapping pairs are unique, and each of the pluralityof medical condition terms includes at least one word. In variousembodiments, a second medical report that is not included in theselected set is received via the network. A first medical condition termis identified in the second medical report. A first alias mapping pairof the plurality of alias mapping pairs is identified by determining asecond medical condition term of the plurality of medical conditionterms that corresponds to the first alias mapping pair comparesfavorably to the first medical condition term. A second medical code ofthe plurality of medical codes that corresponds to the first aliasmapping pair is mapped to a second medical scan corresponding to thesecond medical report.

In various embodiments, the first medical code is a SNOMED code, CPTcode, an ICD-9 code, or an ICD-10 code. In various embodiments, themedical report natural language model is a neural network. In variousembodiments, generating the medical report natural language modelincludes utilizing a forward propagation algorithm on the plurality ofmedical reports to generate a preliminary set of neural networkparameters, and by utilizing a back propagation algorithm to generate anupdated set of neural network parameters based on a calculated set ofparameter errors and the preliminary set of neural network parameters.Determining the first medical code includes utilizing the forwardpropagation algorithm on the first medical report based on the updatedset of neural network parameters.

In various embodiments, utilizing the medical report natural languagemodel to determine the first medical code includes identifying arelevant medical term in the first medical report. The relevant medicalterm and the first medical code are transmitted to a client device viathe network for display by a display device in conjunction with thefirst medical report. The relevant medical term is identified in thenatural language text data of the first medical report in conjunctionwith displaying the first medical code.

In various embodiments, a second plurality of medical reports that arenot included in the selected set are received via a network. At leastone second medical code for each of the second plurality of medicalreports is determined by utilizing the medical report natural languagemodel on the second plurality of medical reports. A training set thatincludes a second plurality of medical scans corresponding to the secondplurality of medical reports is generated. A medical scan image analysismodel is generated based on the training set and the at least one secondmedical code corresponding to each of the second plurality of medicalreports by applying computer vision techniques to the second pluralityof medical scans. A third medical scan is received via the network. Athird medical code of the plurality of medical codes is determined byutilizing the medical scan image analysis model on the third medicalscan. The third medical code is mapped to a third medical scan.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may further be used herein, the term “associatedwith”, includes direct and/or indirect coupling of separate items and/orone item being embedded within another item. As may still further beused herein, the term “automatically” refers to an action causeddirectly by a processor of a computer network in response to atriggering event and particularly without human interaction.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing device” and/or “processing unit” maybe a single processing device or a plurality of processing devices. Sucha processing device may be a microprocessor, micro-controller, digitalsignal processor, graphics processing unit, microcomputer, centralprocessing unit, field programmable gate array, programmable logicdevice, state machine, logic circuitry, analog circuitry, digitalcircuitry, and/or any device that manipulates signals (analog and/ordigital) based on hard coding of the circuitry and/or operationalinstructions. The processing module, module, processing circuit, and/orprocessing unit may be, or further include, memory and/or an integratedmemory element, which may be a single memory device, a plurality ofmemory devices, and/or embedded circuitry of another processing module,module, processing circuit, and/or processing unit. Such a memory devicemay be a read-only memory, random or pseudo-random access memory,volatile memory, non-volatile memory, static memory, dynamic memory,flash memory, cache memory, and/or any device that stores digitalinformation. Note that if the processing module, module, processingcircuit, and/or processing unit includes more than one processingdevice, the processing devices may be centrally located (e.g., directlycoupled together via a wired and/or wireless bus structure) or may bedistributedly located (e.g., cloud computing via indirect coupling via alocal area network and/or a wide area network). Further note that if theprocessing module, module, processing circuit, and/or processing unitimplements one or more of its functions via a state machine, analogcircuitry, digital circuitry, and/or logic circuitry, the memory and/ormemory element storing the corresponding operational instructions may beembedded within, or external to, the circuitry comprising the statemachine, analog circuitry, digital circuitry, and/or logic circuitry.Still further note that, the memory element may store, and theprocessing module, module, processing circuit, and/or processing unitexecutes, hard coded and/or operational instructions corresponding to atleast some of the steps and/or functions illustrated in one or more ofthe Figures and/or described herein. Such a memory device or memoryelement can be included in an article of manufacture. While theprocessing module, module, processing circuit, and/or processing unitdevice may be a general purpose computing device, the execution of thehard coded and/or operational instructions by the processing module,module, processing circuit, and/or processing unit configures such ageneral purpose computing device as a special purpose computing deviceto implement the corresponding steps and/or functions illustrated in oneor more of the Figures and/or described herein. In particular, the hardcoded and/or operational instructions by the processing module, module,processing circuit, and/or processing unit implement acts and algorithmsperformed by the processing module, module, processing circuit, and/orprocessing unit. Such acts and algorithms can be identified by name, canbe illustrated via flowchart and/or described in words.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

The term “system” is used in the description of one or more of theembodiments. A system implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A system may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a system may containone or more sub-system, each of which may be one or more systems.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, random orpseudo-random access memory, volatile memory, non-volatile memory,static memory, dynamic memory, flash memory, cache memory, and/or anydevice that stores digital information. The memory device may be in aform a solid state memory, a hard drive memory, cloud memory, thumbdrive, server memory, computing device memory, and/or other physicalmedium for storing digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. A medical scan diagnosing system, comprising: aprocessing system that includes a processor; and a memory that storesexecutable instructions that, when executed by the processing system,facilitate performance of operations comprising: receiving, via anetwork, a first medical scan; generating first diagnosis data of thefirst medical scan by performing a first medical scan inference functionthat utilizes a computer vision model on the first medical scan;transmitting, via the network, the first medical scan to a first clientdevice associated with a first user of the medical scan diagnosingsystem, wherein the first medical scan is displayed to the first uservia an interactive interface displayed by a first display devicecorresponding to the first client device; receiving, via the network,first review data from the first client device, wherein the first reviewdata is generated by the first client device in response to a firstprompt displayed by the first display device via the interactiveinterface to provide the first review data; determining that the firstreview data indicates that the first diagnosis data is incorrect;generating a plurality of diagnosis data, wherein each of the pluralityof diagnosis data is generated by performing the first medical scaninference function on a corresponding one of a plurality of medicalscans, wherein the plurality of medical scans includes the first medicalscan; determining a subset of the plurality of diagnosis data isincorrect, wherein the subset includes the first diagnosis data;generating updated first diagnosis data based on the first review datain response to the first review data indicating that the first diagnosisdata is incorrect; transmitting, via the network, the updated firstdiagnosis data to a second client device associated with a requestingentity; and automatically generating an updated first medical scaninference function, based on determining the first review data indicatesthat the first diagnosis data is incorrect and further based ondetermining a size of the subset exceeds a predetermined threshold, bytraining the updated first medical scan inference function on a trainingset.
 2. The medical scan diagnosing system of claim 1, wherein theoperations further comprise: receiving, via a network, a second medicalscan; generating second diagnosis data of the second medical scan byperforming the updated first medical scan inference function on thesecond medical scan; transmitting, via the network, the second medicalscan to the first client device associated with the first user of themedical scan diagnosing system, wherein the second medical scan isdisplayed to the first user via an interactive interface displayed bythe first display device corresponding to the first client device;receiving, via the network, second review data from the second clientdevice, wherein the second review data is generated by the first clientdevice in response to a second prompt displayed by the first displaydevice via the interactive interface to provide the second review data;generating updated second diagnosis data based on the second reviewdata; and transmitting, via the network, the updated second diagnosisdata to the second client device associated with the requesting entity.3. The medical scan diagnosing system of claim 1, wherein the operationsfurther comprise transmitting the first diagnosis data to the firstclient device, and wherein the first diagnosis data is displayed to thefirst user via the interactive interface in conjunction with the firstmedical scan.
 4. The medical scan diagnosing system of claim 1, whereinthe first review data includes second diagnosis data in response to thefirst prompt displayed by the first display device via the interactiveinterface to provide the second diagnosis data, wherein the seconddiagnosis data is generated by the first client device based on a blindreview, by the first user, of the first medical scan without review ofthe first diagnosis data, and wherein determining the first review dataindicates that the first diagnosis data is incorrect includes:calculating a consensus confidence score based on a comparison of thefirst diagnosis data to the second diagnosis data; and comparing theconsensus confidence score to a consensus confidence threshold.
 5. Themedical scan diagnosing system of claim 1, wherein the operationsfurther comprise: receiving a second medical scan; generating seconddiagnosis data of the second medical scan by performing the firstmedical scan inference function on the second medical scan; generatingmodel quality check condition data in response to the second diagnosisdata indicating that the second medical scan corresponds to a normaldiagnosis; transmitting the second medical scan to the first clientdevice in response to the model quality check condition data indicatingthat a quality check condition is met, wherein the second medical scanis displayed to the first user via the interactive interface displayedby a first display device corresponding to the first client device;receiving second review data from the first client device, wherein thesecond review data is generated by the first client device in responseto a second prompt displayed by the first display device via theinteractive interface to provide the second review data; and generatingthe updated first medical scan inference function in further response todetermining the second review data indicates that the second diagnosisdata is incorrect.
 6. The medical scan diagnosing system of claim 1,wherein the operations further comprise: selecting the first medicalscan inference function from a plurality of medical scan inferencefunctions based on a first medical scan classifier corresponding to thefirst medical scan.
 7. The medical scan diagnosing system of claim 6,wherein the operations further comprise: generating a new medical scanclassifier by performing a medical scan classifier function on the firstmedical scan; wherein the first medical scan inference function isselected based on the new medical scan classifier in response todetermining that the first medical scan classifier compares unfavorablyto the new medical scan classifier.
 8. The medical scan diagnosingsystem of claim 7, wherein the first medical scan classifier isindicated in metadata of the first medical scan, retrieved from amedical scan database, and wherein the operations further comprise:generating updated metadata corresponding to the first medical scanbased on the new medical scan classifier; and mapping the updatedmetadata to the first medical scan in the medical scan database.
 9. Themedical scan diagnosing system of claim 6, wherein the operationsfurther comprise: determining to split a scan category associated withthe first medical scan classifier into multiple subcategories; andgenerating the updated first medical scan inference function and asecond updated first medical scan inference function based ondetermining to split the scan category, wherein the updated firstmedical scan inference function corresponds to a first one of themultiple subcategories, and wherein the second updated first medicalscan inference function corresponds to a second one of the multiplesubcategories.
 10. The medical scan diagnosing system of claim 1,further comprising a user database that includes a plurality ofregistered entities that includes the requesting entity, wherein thefirst medical scan is received from the second client device, andwherein the operations further comprise: retrieving usage datacorresponding to the requesting entity from the user database inresponse to receiving the first medical scan; and generating the firstdiagnosis data in response to determining the usage data comparesfavorably to a usage quota.
 11. The medical scan diagnosing system ofclaim 1, wherein the operations further comprise: generating testdiagnosis data for each medical scan in a medical scan test set byperforming the first medical scan inference function on each medicalscan in the medical scan test set; generating model quality check databy comparing the test diagnosis data to truth diagnosis datacorresponding to the medical scan test set; and updating the firstmedical scan inference function in response to determining that themodel quality check data compares unfavorably to the truth diagnosisdata.
 12. The medical scan diagnosing system of claim 11, wherein thetest diagnosis data is generated in response to determining an update toat least one of: a hardware component or a software component of atleast one of: the processing system or the memory.
 13. A method forexecution by a medical scan diagnosing system that includes a processor,the method comprising: receiving, via a network, a first medical scan;generating first diagnosis data of the first medical scan by performinga first medical scan inference function that utilizes a computer visionmodel on the first medical scan; transmitting, via the network, thefirst medical scan to a first client device associated with a first userof the medical scan diagnosing system, wherein the first medical scan isdisplayed to the first user via an interactive interface displayed by afirst display device corresponding to the first client device;receiving, via the network, first review data from the first clientdevice, wherein the first review data is generated by the first clientdevice in response to a first prompt displayed by the first displaydevice via the interactive interface to provide the first review data;determining that the first review data indicates that the firstdiagnosis data is incorrect; generating a plurality of diagnosis data,wherein each of the plurality of diagnosis data is generated byperforming the first medical scan inference function on a correspondingone of a plurality of medical scans, wherein the plurality of medicalscans includes the first medical scan; determining a subset of theplurality of diagnosis data is incorrect, wherein the subset includesthe first diagnosis data; generating updated first diagnosis data basedon the first review data in response to the first review data indicatingthat the first diagnosis data is incorrect; transmitting, via thenetwork, the updated first diagnosis data to a second client deviceassociated with a requesting entity; and automatically generating anupdated first medical scan inference function, based on determining thefirst review data indicates that the first diagnosis data is incorrectand further based on determining a size of the subset exceeds apredetermined threshold, by training the updated first medical scaninference function on a training set.
 14. The method of claim 13,further comprising: receiving a second medical scan; generating seconddiagnosis data of the second medical scan by performing the firstmedical scan inference function on the second medical scan; generatingmodel quality check condition data in response to the second diagnosisdata indicating that the second medical scan corresponds to a normaldiagnosis; transmitting the second medical scan to the first clientdevice in response to the model quality check condition data indicatingthat a quality check condition is met, wherein the second medical scanis displayed to the first user via the interactive interface displayedby a first display device corresponding to the first client device;receiving second review data from the first client device, wherein thesecond review data is generated by the first client device in responseto a second prompt displayed by the first display device via theinteractive interface to provide the second review data; and generatingthe updated first medical scan inference function in further response todetermining the second review data indicates that the second diagnosisdata is incorrect.
 15. The method of claim 13, further comprising:selecting the first medical scan inference function from a plurality ofmedical scan inference functions based on a first medical scanclassifier corresponding to the first medical scan.
 16. The method ofclaim 15, further comprising: generating a new medical scan classifierby performing a medical scan classifier function on the first medicalscan; wherein the first medical scan inference function is selectedbased on the new medical scan classifier in response to determining thatthe first medical scan classifier compares unfavorably to the newmedical scan classifier.
 17. The method of claim 15, wherein the firstmedical scan classifier indicates one of a plurality of anatomicalregions, and wherein the plurality of medical scan inference functionscorrespond to the plurality of anatomical regions.
 18. The method ofclaim 15, wherein a user database includes a plurality of registeredentities that includes the requesting entity, and wherein the firstmedical scan is received from the second client device, furthercomprising: retrieving usage data corresponding to the requesting entityfrom the user database in response to receiving the first medical scan;and generating the first diagnosis data in response to determining theusage data compares favorably to a usage quota.
 19. An article ofmanufacture that includes a tangible storage medium that storesoperational instructions, that when executed by a processor, causes theprocessor to: receive a first medical scan from a first client device;generate first diagnosis data of the first medical scan by performing afirst medical scan inference function that utilizes a computer visionmodel on the first medical scan; transmit the first medical scan to afirst client device associated with a first user, wherein the firstmedical scan is displayed to the first user via an interactive interfacedisplayed by a first display device corresponding to the first clientdevice; receive first review data from the first client device, whereinthe first review data is generated by the first client device inresponse to a first prompt displayed by the first display device via theinteractive interface to provide the first review data; determine thatthe first review data indicates that the first diagnosis data isincorrect; generating a plurality of diagnosis data, wherein each of theplurality of diagnosis data is generated by performing the first medicalscan inference function on a corresponding one of a plurality of medicalscans, wherein the plurality of medical scans includes the first medicalscan; determining a subset of the plurality of diagnosis data isincorrect, wherein the subset includes the first diagnosis data;generating updated first diagnosis data based on the first review datain response to the first review data indicating that the first diagnosisdata is incorrect; transmit the updated first diagnosis data to thefirst client device; and automatically generate an updated first medicalscan inference function, based on determining the first review dataindicates that the first diagnosis data is incorrect and further basedon determining a size of the subset exceeds a predetermined threshold,by training the updated first medical scan inference function on atraining set.